If the first post was about dismantling assumptions, this one is about replacing them with certainty. Back then, we had a decoder that worked, a pile of strange patterns, and a growing suspicion that the name RLE was little more than a historical artifact. Now, after walking the bitstream one bit at a time and validating every rule against a real corpus, the format is no longer a black box. It has structure, semantics, and — finally — a specification.

When I started digging into this format, I expected some variation of classic run-length encoding. What I found instead was closer to a tiny virtual machine for pixels. Yes, there are runs, but there are also signed deltas, literal spans, and a single bit-cursor shared across colour channels. Nothing is byte-aligned, nor accidental: every bit participates in reconstructing UI artwork that was never meant to be seen outside its original renderer.

The Turning Point: Transparency

Up until then, decoding felt linear: read tokens, expand pixels, move forward. Alpha changed the rules. Transparent spans emit pixels without consuming colour data. Buffers persist across segments; the “previous value” never truly resets. Suddenly the format stopped behaving like a simple stream and revealed itself as a stateful decoding model layered on top of a bit-level language. Once that behaviour became clear, the rest of the structure — row navigation, token width B, palette post-processing — just snapped into place.

Verification

Hundreds of files were decoded and compared pixel-for-pixel against the original renderer. No heuristics, no guesses — just reproducible behaviour. At that point, the reverse engineering effort had effectively crossed a boundary: the goal was no longer to make a decoder work, but to describe why it works.

So instead of shipping another tool, I wrote the document I wish I had at the beginning: a complete, clean-room specification of the format. It defines the container, the payload, the bitstream grammar, channel interleaving, the alpha meta-decoder, palette semantics, DPI scaling, and the precise arithmetic rules that govern decoding. Every field is explained, every algorithm is explicit. Every edge case that once looked like noise now has a name.

The format is no longer folklore, no longer that “strange Messenger blob.” It is a fully described image format with a deterministic decoding model — and you can read it here:

RLE-DIB Format Specification

The mystery is over. From here on, this format is something one can implement, reason about, and build against — and, perhaps most satisfying of all, the decoder is now far smaller than the puzzle that led to it. I validated the spec by implementing a Swift parser — aside from the Python parser included in the spec itself — that faithfully rendered assets on macOS.

Legal & Research Disclaimer

This material is intended strictly for academic research, reverse engineering for interoperability, and software preservation.

The author does not distribute, reproduce, or provide any proprietary software, including components originally developed by Microsoft.

This work is independent, clean-room in nature, and is not affiliated with, endorsed by, or supported by any original vendor.

Any trademarks, formats, or software referenced remain the property of their respective owners.

The first step to make this way easier is to enable logging for troubleshooting, which happily dumps really interesting stuff into ~/Library/Logs/Microsoft-Messenger.log.

The Login Dance

Once an email and password is provided, the client immediately opens a raw TCP connection to messenger.hotmail.com, port 1863. This seems to be the server that speaks the MSNP protocol, given the very first message from the client:

VER 1 MSNP16 MSNP15 MSNP14 MSNP13 CVR0

The protocol is fairly simple: text-based, each message ending in CR-LF. It has the following format:

  VER 1 MSNP16 MSNP15 MSNP14 MSNP13 CVR0
  ━┳━ ┳ ━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
   │  │  │
   │  │  └──▶ Payload
   │  └──▶ Message ID
   └──▶ Operation

The message begins with an operation code, followed by the message ID, and a payload. The server always responds with the same message ID, and a corresponding operation code and payload.

The VER operation is a handshake between the client and server. The client announces the protocols it supports, and the server answers with one of the protocols it supports. So the fake server can just pick the highest one and be done with it.

I suppose CVR stands for two different things during the flow. When sent by the client, it may be a Client Version Report, meaning “I use the base CVR capability schema, revision zero”. For the server reply, CVR may be a Capability Vector Revision, but I can’t confirm that. Echoing the CVR0 back to the client is enough to make it happy.

This leads to the following response from the server:

VER 1 MSNP16 CVR0

Once the version is agreed between parties, the communication continues with the client sending:

CVR 2 0x0409 mac 10.14.6 i386 macmsgs 8.0.1 macmsgs hey@vito.io

This is the Client Version Report, containing the following fields:

• 0x0409: Microsoft’s LCID (Locale ID) in Hex. 0x0409 is simply en-US; Italian, for instance would be 0x0410.
• mac: Platform/OS family.
• 10.14.6: OS version string.
• i386: Architecture. This seems to be hardcoded in the binary.
• macmsgs: Client product identifier. Just identifies as Microsoft Messenger for Mac.
• 8.0.1: The client version.
• macmsgs: Again. Not sure why.
• hey@vito.io: User identity. That’s me!

Even though the client waits for a response, what the server returns doesn’t seem to be used at all. My theory is that this may be just telemetry, or a mechanism for Microsoft to block old clients or architectures.

For now the fake server is replying with CVR 2 1 2 3 4 5, and Messenger is happy. After getting the reply, the client then starts the real authentication flow:

USR 3 TWN I hey@vito.io

From observation, USR is the User Authentication OP. Decoding it:

• TWN: I assume this means Ticket With Nexus, considering what happens later.
• I: Seems to be the authentication phase the client is currently at. I may indicate Initiate? Initial?
• hey@vito.io: That me, again! This just includes whatever the user inputs in the login “Email” field.

The server then replies with some metadata that’s opaquely handled by the client and appended to an HTTP Authorization header. So the server can reply with literally anything:

USR 3 TWN S fake=1

Then things shifts a little:

The Passport Backend

While still connected to the TCP server, the client opens a separate connection entirely – a concurrent, out-of-band flow to the defunct Passport.net server. Not without caveats, though.

All communication with Passport.net uses TLS – Actually, not TLS. SSL. – A really old one: SSLv2 Hello, TLS 1.0. And the best part is that OpenSSL dropped that a while ago. What I need now is a CA Root installed on Mojave, and a TLS terminator so I can continue observing traffic.

So I got a certificate for nexus.passport.com, and wrote a little TLS shim in C to route TLS traffic from Mojave to my alpine box running the fake servers. Just a matter of compiling OpenSSL 1.0.2u, accepting the connection with the right certificate, and proxying the raw TCP data. OpenSSL was linked statically with the final binary, so we don’t break the whole OS.

The first request is a GET to /rdr/pprdr.asp. With some guesswork we can assume rdr stands for Redirect or Redirector, and pprdr may stand for Passport Redirect(or) – naming those things don’t really matter, but it’s nice to give names to those pesky abbreviations Microsoft uses.

The response has no body, but the client expects an HTTP header named PassportURLs with a few fields:

• DARealm: The realm name. Not used by the client, but it requires it to be present.
• DALogin: The realm host and login path. Microsoft seemed to use nexus.passport.com/login2.srf. The lack of schema is intentional, the client will use HTTPS. If you are wondering, DA stands for Domain Authentication.
• Properties: Not used by the client, but it requires it to be present.

Those pairs get concatenated as K=V separated by commas.

Then, the actual login magic happens. The client POSTs to DALogin without a body, but with an interesting header:

Authorization: Passport1.4 fake=1,OrgVerb=GET,OrgURL=http%3A%2F%2Fmessenger.msn.com,sign-in=hey%40vito.io,pwd=asdf

Breaking it down:

• Passport1.4: The authentication mechanism version. Hardcoded.
• fake=1, the opaque value provided by the TCP server on the S phase of the TWN operation. Everything connects!
• OrgVerb=GET: I honestly have no idea what that’s supposed to be. Maybe something for federation, or to allow Messenger servers outside Microsoft?
• OrgURL=http://messenger.msn.com: Same as OrgVerb.
• sign-in=hey@vito.io: The identity of the user being authenticated
• pwd=asdf: The password provided by the user on the login window.

The response can again have an empty body, but the client expects a specific header to be present:

Authentication-Info: Passport1.4 da-status=success,from-PP='a=HENLO'

Breaking it down:

• da-status: The status returned from the Domain Authentication. Here we are indicating the authentication was successful.
• from-PP='a=HENLO': Assuming this as From Passport. The value is opaque to the client, it just returns this to the TCP server.

This is the last interaction with Passport – for now! The dance continues with the TCP server again, where the client sends the following message:

Back to TCP

USR 4 TWN S a=HENLO 235411cb78234a468feb6f1bccca43fd

Same OP, different message ID – as it’s a new message, – and the value returned from passport is handled opaquely and just returned to the TCP server. The final hex is 32 characters – MD5-length – but I haven’t confirmed what it represents. Correlation token, telemetry, session nonce; take your pick.

With this, the TCP server can just FINALLY complete the authentication flow by replying with:

USR 4 OK hey@vito.io 1 0

• OK: Authentication passed. Hooray.
• hey@vito.io: The user’s identity
• 1: Authentication flags. 1 seems to be enough to keep the client happy.
• 0: Capability flags. Just echoing from CVR.

Should we expect something else over TCP now? Absolutely not. The plot thickens:

Microsoft SOAP Services

Microsoft loves SOAP for reasons one can’t even reason about. As I’m just trying to boot the app, and not actively use it, the idea is to just provide the minimal amount of stuff to make it get to the main window post-login.

Looking at how the client uses those endpoints, I can again just provide what it deems required. One interesting thing to notice, however, is that for each SOAP endpoint, the client seems to perform authentication once again against the Passport service, and relays whatever the authentication endpoints provides in response to a TicketToken field.

SOAP goes – at least initially – to one endpoint:

byrdr.omega.contacts.msn.com receives a POST request for /abservice/SharingService.asmx, with an HTTP header SOAPAction: http://www.msn.com/webservices/AddressBook/FindMembership, followed by another POST, this time to /abservice/abservice.asmx, with the same header, but this time containing the value http://www.msn.com/webservices/AddressBook/ABFindAll.

On those, FindMembership seems to be responsible to indicate the membership graph. The MSN Address Book service (aka abservice) has several logical lists, where each one corresponds to a different kind of “membership” in a materialised view for the current user. Those lists are:

• FL: The Forward List, representing one’s contacts (people one added)
• AL: The Allow List, representing who can see one’s presense status.
• BL: The Block List, containing blocked contacts.
• RL: The Reverse List, containing who added the user (reverse of FL).
• PL: The Pending List, containing pending invitations.

FindMembership provides the AL, BL, and RL lists, while ABFindAll returns what’s enough for the client to rebuild the FL list.

ABFindAll is a different story; it returns the full contact records for the authenticated user. Each record contains several fields, including:

• contactId: a GUID representing the contact.
• contactType: the contact’s type. Regular is the only value that’s of my interest right now.
• passportName: the user’s Passport name. Basically their email.
• displayName and quickName: the user’s display name. Not sure why it is sent twice.

The object may contain lots of other fields, but those are the most relevant to achieve our objective.

Once those SOAPs are returned, the client resumes communication with the TCP server:

The Final Dance

The next operation requested by the client is a new kind we haven’t seen yet: a two-part operation. This is what is sent:

ADL 5 245\r\n
<ml l="1"><d n="local"><c n="away" l="3" t="1" /><c n="brb" l="3" t="1" /><c n="busy" l="3" t="1" /><c n="hidden" l="3" t="1" /><c n="idle" l="3" t="1" /><c n="lunch" l="3" t="1" /><c n="online" l="3" t="1" /><c n="phone" l="3" t="1" /></d></ml>

This will need some explanation. Through the previous SOAP, I’m setting a few users to have each one showing a different status. Here are the users:

  GUID                                 Passport
  ------------------------------------ -------------
  11111111-1111-1111-1111-111111111111 online@local
  22222222-2222-2222-2222-222222222222 busy@local
  33333333-3333-3333-3333-333333333333 idle@local
  44444444-4444-4444-4444-444444444444 away@local
  55555555-5555-5555-5555-555555555555 brb@local
  66666666-6666-6666-6666-666666666666 lunch@local
  77777777-7777-7777-7777-777777777777 phone@local
  88888888-8888-8888-8888-888888888888 hidden@local

ADL is the Address List command. The payload 245 indicates how many bytes follow after the terminator (\r\n), comprising a second payload, that’s formatted below:

<ml l="1">
  <d n="local">
    <c n="away"   l="3" t="1" />
    <c n="brb"    l="3" t="1" />
    <c n="busy"   l="3" t="1" />
    <c n="hidden" l="3" t="1" />
    <c n="idle"   l="3" t="1" />
    <c n="lunch"  l="3" t="1" />
    <c n="online" l="3" t="1" />
    <c n="phone"  l="3" t="1" />
  </d>
</ml>

The XML contains a single ml (Membership List) element with an l attribute with value 1, indicating this is the first list exchanged through the ADL OP. Each sub-list is split by domain, and since all my users are @local, we get a single d (Domain) element, with the n attribute set to local. Inside we have each user as a c element, their name in the n attribute, l (List) set to 3, which is a bitmask indicating that the contact is both on FL and AL lists and t (Type) set to 1, indicating a Passport-type identity.

The bitmask values are:

• 1 = Forward List (FL) – contacts the user added
• 2 = Allow List (AL) – contacts allowed to see the user’s status
• 4 = Block List (BL)
• 8 = Reverse List (RL)
• 16 = Pending List (PL)

For now we just acknowledge the message, so it can continue with its process:

ADL 5 OK\r\n

The next OP we get from the client is PRP:

PRP 6 MFN hey@vito.io

PRP stands for Profile Property. The client is requesting a property to be set: MFN, also known as My Friendly Name. Again, we just need to acknowledge it, but this time with an ACK OP:

ACK 6 PRP\r\n

The client continues with:

BLP 7 AL

BLP is the Blocking Policy. It controls the default privacy rule for people not explicitly on AL and BL. The last token we get is the policy to be applied, which can be one of:

• AL: Allow by default. Unknown users can see and contact the current user.
• BL: Block by default. Only allowed users can see and contact the current user.

The client is setting the policy to Allow. Ok. We just acknowledge it again:

ACK 7 BLP\r\n

Next up: Status update. The client sends:

CHG 8 NLN 1085276192:131088

CHG requests a status update, NLN indicates Online, and is followed by an optional capability bitmask (two numbers, colon-separated). For now I’ll ignore those, but they will be important in the future, depending on how this progresses.

For this one we will echo everything back to the client:

CHG 8 NLN 1085276192:131088\r\n

But this change is also an important trigger for the TCP server. From this point onwards, the client accepts the Initial Status List, (ILN) which assigns statuses and Display Names for each one of its contacts, followed by an UBX OP that provides extra information regarding their profiles, such as a Personal Status Message, and Media Status.

Here we reuse the Message ID from CHG to send one ILN for each contact, along with an UBX. Notice, however, that UBX takes no Message ID. From the previous contact list, I’ll also assign a random UUID for their PSMs:

ILN 8 NLN online@local 1 Online%20User 1085276192 %3Cmsnobj%2F%3E\r\n
UBX online@local 1 89\r\n
<Data><PSM>0b48cf1c-b9a5-4793-b8d4-a8ddd6b13c6c</PSM><CurrentMedia></CurrentMedia></Data>

ILN 8 BSY busy@local 1 Busy%20User 1085276192 %3Cmsnobj%2F%3E\r\n
UBX busy@local 1 89\r\n
<Data><PSM>72b6dca4-811b-4c6a-8b2a-c87cfa6e3299</PSM><CurrentMedia></CurrentMedia></Data>

ILN 8 IDL idle@local 1 Idle%20User 1085276192 %3Cmsnobj%2F%3E\r\n
UBX idle@local 1 89\r\n
<Data><PSM>88360ef7-5152-4ada-8bc8-e0343736e830</PSM><CurrentMedia></CurrentMedia></Data>

ILN 8 AWY away@local 1 Away%20User 1085276192 %3Cmsnobj%2F%3E\r\n
UBX away@local 1 89\r\n
<Data><PSM>d430617a-17bf-4437-a8f2-54aa4d33444e</PSM><CurrentMedia></CurrentMedia></Data>

ILN 8 BRB brb@local 1 Brb%20User 1085276192 %3Cmsnobj%2F%3E\r\n
UBX brb@local 1 89\r\n
<Data><PSM>ecd12daf-0c11-445f-b1ae-4751a509c25c</PSM><CurrentMedia></CurrentMedia></Data>

ILN 8 LUN lunch@local 1 Lunch%20User 1085276192 %3Cmsnobj%2F%3E\r\n
UBX lunch@local 1 89\r\n
<Data><PSM>9bd609bc-d2f8-4e1a-be72-06804bf73183</PSM><CurrentMedia></CurrentMedia></Data>

ILN 8 PHN phone@local 1 Phone%20User 1085276192 %3Cmsnobj%2F%3E\r\n
UBX phone@local 1 89\r\n
<Data><PSM>b1482981-ad81-4a2a-872f-ba3331c6bb91</PSM><CurrentMedia></CurrentMedia></Data>

The ILN command has the following parameters after the Message ID:

• The status (NLN, BSY, IDL, AWY, BRB, LUN, PHN)
• Their email address
• Their Network ID (1 = Passport/Windows Live)
• Their display name, URL Encoded
• Their capabilities flag. Here I’m sending the same caps the client provided to the server, except that it only takes the first portion (before the colon).
• A msnobj payload. Here it is empty, and has no significance to my objective.

UBX has the same format as UUX (see below).

As a side note, the required msnobj portion was what took most of the time during this session. Without it the client just refused to assigned statuses and failed silently. Codepaths leading to this portion were also quite complicated, so getting to the point “oh, that’s what’s missing” took a while.

Then, more state is sent by the client:

UUX 9 53
<Data><PSM></PSM><CurrentMedia></CurrentMedia></Data>

UUX (User eXtended data) is a set of metadata announced by the client. Here it’s setting the PSM (Personal Status Message), and the CurrentMedia to empty values. Again, we just acknowledge it:

ACK 9 UUX\r\n

Following, we get two extra UUX messages:

UUX 10 75\r\n
<EndpointData><Capabilities>1085276192:131088</Capabilities></EndpointData>

and

UUX 11 92\r\n
<PrivateEndpointData><EpName>Rhodes</EpName><ClientType>1</ClientType></PrivateEndpointData>

The first one contains the same capability set we got during the CHG OP. The second is part of the “multi-endpoint” implementation that was being rolled, allowing users to sign in in multiple places at the same time. It contains the endpoint name EpName, that’s carrying the name of the computer running Messenger (Rhodes), and a client type. We can assume 1 represents a Desktop.

Then, the client continues asking for a service URL using the URL OP:

URL 12 MOBILE

There it is asking specifically for the Mobile Messaging / SMS Service MSN used to support. For now, let’s just route it to our server:

URL 12 MOBILE http://10.0.10.126:8080/\r\n

And continues with further URL ops:

URL 13 PROFILE 0x0409
URL 14 CHGMOB

PROFILE is requesting the profile webpage/service URL for a LCID in hex, again, 0x0409 is en-US. CHGMOB is the Change Mobile settings URL, part of the MOBILE capability. For both we will reply the same URL as MOBILE, as the client wouldn’t care anyway.

Followed by:

UUN 15 hey@vito.io 8 9
ignore me

I think UUN is User Update/Notification the value 8 is unknown, and 9 represents the amount of bytes following. For some reason, the client is sending ignore me. Weird. After some Trial-and-Error, the client is happy with a simple ACK:

ACK 15 UUN\r\n

And finally, the client rests sending periodic PNG (Ping) OPs:

PNG 16\r\n

QNG 60\r\n

The result is nothing less than satisfying.

And there it is. Eight contacts, eight statuses, no Microsoft infrastructure involved. A fake TCP server, a TLS shim, a handful of SOAP responses, and a lot of trial and error – enough to convince a 2008 Mac app it’s talking to servers that haven’t existed for over a decade.

That was just a little detour try to relax from the amount of Assembly and C/C++ on Windows. Did it work? No, I had to disassemble the macOS binary, but at least that is known territory.

Later on I will continue on the main project.

But this one… This one lied to me in the first three bytes.

52 4C 45   →   "RLE"

It was not RLE.

Not byte-aligned runs, not (count, value), not even something you could generously call RLE-like. What sits behind that header is a bit-packed predictive codec with interleaved channels, delta predictors, a meta-decoder layered above colour decoding, and a rendering pipeline buried under more code than anyone should have to read for a small icon.

What you are reading is about how I went from opaque binary blobs to a bit-exact decoder, validated against the original renderer. No Microsoft code used. Only observation, instrumentation, patience, and stubbornness.

This is the first step toward a side project I was thinking about for way too long: recreating the Windows Live Messenger 2009 look-and-feel. Same visuals, entirely new engine, open, secure, cross-platform. Not nostalgia, not preservation. Reconstruction.

The most important part of this endeavour would be recreating the assets, or at least extracting them. WLM2009 had stunning visuals that made it look great, both on PCs and Macs. So the first task was set: Let’s get those assets. That couldn’t be hard, right? Right?

Where are the assets?

The UI clearly contained intricate elements: icons, frames, gradients, control visuals, but there were no images anywhere. One may expect to at least find PNGs, BMPs, and such. But no PNG, no BMP, no JPEG, nothing that looked like pixel data, nothing in the resources that resembled bitmaps. It was madness! How could that amount of transparency and effects be nowhere?

Well, let’s make a small detour to a quick history lesson: Microsoft’s internal UI frameworks.

Everything on WLM2009, and actually, in all Windows Live™ products of the same era used an internal framework called DirectUI. After some research, it seems that while people were suffering with MFC and WPF, Microsoft was using a custom UI framework to make Windows Live, Office, and as far as I understood, parts of Windows as well. That was DirectUI. People seem to think they gave up on that, but I haven’t looked deeper, sorry.

So that meant that all that fancy stuff was being drawn by something else, but as WLM also worked on Windows XP, Windows Vista, and Windows 7, binaries should provide those facilities. Awesome, but where are the assets?

Interestingly, what those binaries lack in PNGs and other image formats, they compensate with lots of blobs containing the same three-byte magic header: RLE – Great! Everyone (okay, not everyone) knows RLE stands for Run-Length Encoding, so decoding that would be easy: extract the RLE blob, strip the header, run the algorithm, and we should get some kind of image.

No. Not this time. Doing this yielded nothing but garbage. Again, wrong assumption. Now it feels like even Occam was also lying to me; a big conspiracy.

Jokes aside, one thing is clear: that was no RLE, but something else! Something internal that only Microsoft knew, and that poked that small piece of my brain that really, really likes to dig into this kind of thing – so it took the wheel. Entropy analysis, heuristics, pattern matching, anything that could give any kind of direction on what those blobs were about. Nothing.

That was the moment the situation became clear. These were not compressed images. These could only be the images. There was no fallback, no original bitmap stored somewhere else. The blob itself was the canonical representation of the artwork. But if we cannot decode it, we cannot reproduce the UI. There is no other source of truth.

Finding the real decoder

Searching for RLE on the binaries leads to a forest of misleading symbols. Helpers, wrappers, palette code, rendering glue, dead paths, UI plumbing. Most of those never touch pixels. They prepare, route, or post-process, but they do not decode.

Ghidra helped map the topology, but the generated C was often fiction. Registers reused in unexpected ways confused the decompiler, calling conventions were occasionally guessed wrong, stack frames collapsed, and control flow flattened into something that looked plausible but was not what the CPU executed. The assembly was the only reliable source of truth.

So the process became mechanical: Ghidra to understand structure, x32dbg to observe reality, follow registers and observe the process’s memory, confirm assumptions against execution, and treat decompiled C as a hint rather than documentation; instead of trusting the C, the only source of truth was the generated assembly from the binaries. And that hit hard.

But no problem, nothing we can’t solve, right? Eventually one function stood out: RLEDrawToDIB. That’s the one responsible for converting the mysterious RLE into something observable, and even better, on a DIB (Device-Independent Bitmap), which can be exported.

Ok, we have the entrypoint, but what now?

Building an Oracle

The main executable helped a lot finding where and what was responsible for drawing, but the amount of noise after that was just too much, after all, the main binary did way more than just rendering. x32dbg was constantly pausing execution at way too many places, and analysing the stack at those points became tiring.

The plan now was to devise an oracle. Its job is to exercise only the hot path concerning parsing and drawing. For that, one must first find the entrypoint – which is already done, we have RLEDrawToDIB, – and make it work isolated from the host binary. As it may be known, dear reader, I am no Windows developer; most of my life I used Linux, Unix, OS X, and now macOS, so this is a whole new world. But as I usually say, in the end it’s just code, so let’s jump in and understand how to make this work.

The idea is dumb in the best way: load uxcore.dll into a process we control, resolve internal functions by raw offset – no exports, no symbols, just hard RVAs pulled from Ghidra. Reconstruct just enough of the calling sequence to drive the real renderer into a DIB section we own. Then dump every pixel.

The shim is small: a macro computes function pointers as base + offset into the loaded binary. A handful of typedefs describe the calling conventions Ghidra helped identify, a thin wrapper mirrors the original dispatch logic, including the nine-slice path, and routes execution through the same code paths the real UI would take. The renderer writes into a top-down 32bpp DIB section, BGRA in memory, which gets converted to RGBA and dumped as raw bytes alongside a JSON sidecar with dimensions, pixel format, and colour-source metadata.

That raw RGBA dump is the oracle. Not the PNG, as PNGs go through GDI+ encoding and are useful for eyeballing, but byte-level comparison needs raw pixels. Every single byte my clean-room decoder produces can now be diffed against what the original renderer actually drew. No ambiguity, no interpretation, no room for “close enough.”

But the oracle was not just a validation tool, but an exploration tool. With the real renderer running under our control, we can instrument it, set breakpoints, watch which functions get called and in what order, and trace the actual execution path through those functions. Instead of staring at a forest of dead code in Ghidra trying to guess what matters, we can observe what matters. The oracle narrowed the search space from everything to exactly this.

The process ran across 516 files exported from Messenger. For each one: load the blob, ask uxcore.dll for the frame size, allocate a DIB, clear to transparent, render, flush, dump. A system-colour table gets serialised at the end so palette-mode files can be validated deterministically. The whole thing is a few hundred lines of C++ and the ugliest #define you have ever seen, but it worked.

A note aside, outputting PNG using GDI+ is ugly as ugly gets, not to mention Windows ABI. If there’s a hell, I’ve found it. I’m also skipping a whole part where the oracle crashed, memory was corrupted, NULL pointers were dereferenced, and everything that could go wrong, did. Just for context, here what I tried once it refused to work:

• Do we need to initialize COM for this process? Maybe. Would it hurt if we did it? Hardly. So COM initialization was done. As the oracle was a “Console Application”, in Windows lingo, COINIT_APARTMENTTHREADED seemed enough. That didn’t change anything.
• Maybe Common Controls were missing? Did the DLL expect this to be ready? InitCommonControlsEx to the rescue. Guess what? Nothing changed.
• Maybe GDI+ needed to be initialized? GdiplusStartup was called, and guess what? That wasn’t it either.

Only then I took the time to look at the host binary again, only to find two interesting functions being called on uxcore.dll: UXCoreInitProcess and UXCoreInitThread. And ed ecco! No crash.

This exposed the whole hot path the decoder used: every function could now be isolated, inspected, and observed. “But how many?” you may be asking yourself, and I’m glad you asked. From there the hot decoding path expanded into one hundred and twenty-one internal functions, excluding Win32 calls. That was the actual renderer. Now to get the gist of the format, we just need to extract all of them as both C code and assembly. The result? ~20k SLOC of C code, ~41k of Assembly. I hoped I had enough coffee in the kitchen for that and for the blessing of Our Lord Turing, cause I knew that would hurt. Not that Assembly is a problem, but after optimisations and some weird stuff we know Microsoft compilers do, I knew I was signing up for headache and frustration.

Ghidra vs reality

One may wonder why export both C and Assembly. Well, at several points, the decompiler confidently produced code that looked reasonable and also was completely wrong. And not wrong in subtle ways, but wrong in ways that obliterate decoding on the first byte. Some examples (we will get to each of these):

• Bit order: The decompiled logic strongly suggested MSB-first reading. The stream is LSB-first. That single inversion produces garbage immediately, and it is the kind of mistake that looks correct on paper until you watch actual execution and realise every bit is landing in the wrong position.
• The predictor: Decompiled output implied frequent resets. In reality, it carries across runs within a row. That persistence is exactly why gradients compress well, and why breaking it turns smooth ramps into staircases.
• Row navigation: The decompiled view made it look absolute. It is forward-relative, with row zero starting at the last index entry. Miss that, and the decoder walks into the middle of someone else’s bitstream.
• Alpha: The reconstructed C placed it inline with colour. In execution, it sits above colour decoding, as a meta-layer that controls when colour tokens are consumed. Without modelling that correctly, colour buffers desynchronise and pixels drift. This one took the longest to untangle.
• Token alignment: Loops in the pseudocode looked byte-aligned. They are not. Everything is bit-aligned. Every. Single. Field.

For those starting in reverse engineering: The lesson is simple, but important. Decompiled code is not truth. Execution is.

The Trampoline

The format remained opaque until a small piece of assembly changed everything. There was a dispatch trampoline routing execution into different meta-decoders depending on header flags. That explained why streams behaved differently even when they looked similar.

Some used palette. Some used alpha runs. Some consumed colour continuously, others stalled. Some interleaved channels, others did not. The structure finally emerged. This was a bitstream, not a byte stream. Channels decoded independently. Predictive delta replaced repetition. Alpha was not inline; it existed in a meta-layer. Rows were not sequential but navigated via a pointer table. Bits were read least-significant-bit first.

This was not RLE. This was a “small” image codec wearing an RLE badge.

The Format

With the oracle, the trampoline, and 41k lines of assembly in front of me, the format started to take shape. What follows is a high-level overview, enough to understand why calling this “RLE” is an insult to the codec. A full specification with worked examples exists, but that deserves its own post.

The Header

Every blob opens with six bytes. Three magic bytes (RLE), an image count, a flags byte encoding the tile topology and payload size width, and a DPI byte. Six bytes. That is the entire file header.

The tile topology is a 4-bit nibble packed into the flags byte. Each bit controls whether a fixed border exists on that edge — left, top, right, bottom. The result is a nine-patch system: a 1x1 grid when no bits are set, a full 3x3 when all four are. Sixteen possible configurations, all encoded in half a byte. After some research, I noticed Android also have those nine-patch PNGs! Same idea, different decade. Microsoft got there first. (Is that a first?)

Sub-Images

Each sub-image carries a 4-byte header dword packed with geometry and flags. Width and height, yes, but also: palette mode, 5-bit or 8-bit precision, alpha presence, and the width of the row-navigation index entries. Thirteen bits for width, twelve for height, and the rest is flags. Every bit accounted for.

The row-navigation index is where things get even weirder. Rows are not stored sequentially. A table of forward-relative offsets sits at the start of the pixel data, and row zero begins at the position of the last index entry. It is elegant once you see it, and completely opaque until you do.

The Bit-Cursor

Everything past the row index is a bitstream. Not bytes. Bits. Read LSB-first within each octet, shared across all channels in a row. The decoder maintains a bit-cursor: a byte pointer and a bit offset, and every field, from token types to run counts to delta widths, is read through it. Nothing is byte-aligned unless it happens to be by coincidence.

Tokens

Each colour token starts with a 2-bit type and a B-bit run count, where B is derived from the image width. We have four token types:

• Solid fill: one value, repeated.
• Delta-positive: each pixel is the previous value plus a small positive offset.
• Delta-negative: same, but subtracted.
• Literal: raw values, one per pixel.

The deltas are the reason this format compresses well. Gradients, anti-aliased edges, smooth colour transitions: they all become tiny increments instead of full pixel values. The predictor carries across tokens within a row, which is what makes it work. And what makes it break if your decompiler tells you it resets. I’m looking at you, Ghidra.

The Alpha Meta-Decoder

And then there is alpha. It does not sit alongside colour: it wraps it.

A 2-bit alpha type is read before each group of pixels: transparent run, opaque run, or explicit alpha. Transparent runs emit zeros and skip colour decoding entirely. Opaque runs consume colour normally. Explicit alpha decodes an alpha token first, then pulls the corresponding colour values from the channel buffers.

This meta-layer is why the format was so hard to reverse. Colour and alpha share the same bitstream, the same cursor, but alpha controls when colour is consumed. Get that wrong and everything desynchronises. Ghidra placed it inline, and guess what? It is not inline. It is a state machine sitting above the channel interleaver.

Validation

With the format understood and a clean-room decoder written in Python (no dependencies beyond the standard library) it was time to answer the only question that matters: does it match?

The oracle produced raw RGBA dumps for 516 files extracted from Messenger’s binaries. The decoder ran against the same inputs. Byte-level comparison. No thresholds, no visual inspection, no “close enough.”

516 files. 380 single-tile, 136 multi-tile nine-slice. 516 exact matches. Not a single byte off.

What’s Next

This post covered the archaeology: finding the format, building the tooling, understanding the structure. The next one will go deeper: a full worked decode of a real file, bit by bit, token by token, with annotated hex dumps and the complete specification. If you enjoy reading binary formats the way some people enjoy crossword puzzles, you will like it.

There is also the matter of the DPI scaling pipeline and the palette system, both of which deserve their own attention. And eventually, the actual goal: using all of this to reconstruct the Windows Live Messenger 2009 interface, from scratch, on modern platforms.

But that is for another day.

New Findings

In the last post, I mentioned that I had omitted a few DHCPACK and DHCPINFORM packets, assuming they weren’t particularly important. My DHCP knowledge is a bit rusty, and I naïvely thought: “DHCPINFORM? The client already has an IP. No big deal.”

That assumption was completely wrong.

What was actually happening is that the Ray was requesting someone to inform it about some knobs so it could finish bootstrapping and start the boot sequence! That should have been obvious, especially given that it was retrying the same packet over and over since no one was answering it, but yes, I missed that. Oops!

Now, back on track. The packet clearly indicated what it was looking for:

• Option 53 (DHCP Message Type): DHCPINFORM (0x08)
• Option 55: Parameter Request List
  • Interface MTU (0x1A)
  • X Window System Display Manager (0x31)
  • TFTP Server Name (0x42)
  • Vendor-Specific Information (0x2B)

The choices there are interesting indeed! MTU makes sense, but, XDM? Odd at least.

At this point, we know that:

• The Sun Ray successfully acquires an IP
• It explicitly requests vendor-specific parameters
• It stalls indefinitely without them

Looking for some information on the Web I could find some piece of documentation that Oracle haven’t burned yet, and oh my, it’s telling. For pure archival purposes, I’m mirroring it here: Sun Ray Client Initialization Requirements Using DHCP, but what’s interesting there is the Alternate Vendor-Specific DHCP Options section, listing exactly what we need. The response for the DHCPINFORM SHOULD carry a bit of extra information in the Vendor-Specific Information TLV, but the Ray could piggyback on “Standard” DHCP Options, if your DHCP server didn’t support them. Clever, very Sun-ish.

What’s relevant for us is this: We could add a Vendor-Specific Information to the DHCPACK of the DHCPINFORM packet, and the Ray would just use it. And we also have A LOT of knobs that can be turned. Below is a quick summary:

• AuthSrvr, Type: IP - Mandatory. Single Sun Ray server IP addresses.
• AuthPort, Type: NUMBER - Sun Ray server port.
• NewTVer, Type: ASCII - Desired firmware version.
• LogHost, Type: IP - Syslog server IP address.
• LogKern, Type: NUMBER - Log level for kernel.
• LogNet, Type: NUMBER - Log level for network.
• LogUSB, Type: NUMBER - Log level for USB.
• LogVid, Type: NUMBER - Log level for video.
• LogAppl, Type: NUMBER - Log level for firmware application.
• NewTBW, Type: NUMBER - Bandwidth cap, value is bits per second.
• FWSrvr, Type: IP - Firmware TFTP server IP address.
• NewTDispIndx, Type: NUMBER - Obsolete. Do not use.
• Intf, Type: ASCII - Sun Ray server interface name
• NewTFlags, Type: NUMBER - Obsolete. Do not use.
• AltAuth, Type: IP - List of Sun Ray server IP addresses.
• BarrierLevel, Type: NUMBER - Mandatory. Firmware Download: barrier level.

Lots of interesting stuff, but let’s begin with what can really help us: Logging. It seems the Ray is capable of emitting log data to a Syslog server, and we can set the level for several facilities, including firmware application, network, and kernel! That’s just awesome! Thanks, Sun!

Now, given it wants a Syslog server, one can assume that the log levels are also standardised to the levels Syslog supports, and one would be absolutely right. From RFC 3164 - The BSD syslog Protocol, page 9 contains Table 2. syslog Message Severities, listing from 0 (Emergency) up to 7 (Debug). I think the box may use 6 (Informational) as default, so setting it to 7 may make it be more chatty!

Encoding Vendor-Specific Information

The archived page also explains with less-than-desired details the encoding format, but it’s nothing more, nothing less than a plain old TLV structure. Adorable!

The structure is composed of repeated TLVs: one byte for the tag, one byte for the length, followed by n bytes of value. The only important thing to keep in mind is that any IP type is a simple representation of the IP octets, meaning that 192.168.100.1 becomes 0xC0, 0xA8, 0x64, 0x01.

Now let’s make the Ray talk to us through syslog, and make it chatty. The DHCP Option payload for Vendor-Specific Information would then be:

• Option 0x2B:
  • 0x18, 0x04, 0xC0, 0xA8, 0x64, 0x01. (LogHost, 4 bytes, 192.168.100.1)
  • 0x25, 0x01, 0x07. (LogKern, 1 byte, value 7)
  • 0x26, 0x01, 0x07. (LogNet, 1 byte, value 7)
  • 0x27, 0x01, 0x07. (LogUSB, 1 byte, value 7)
  • 0x28, 0x01, 0x07. (LogVid, 1 byte, value 7)
  • 0x29, 0x01, 0x07. (LogAppl, 1 byte, value 7)

I don’t want to bring up a full syslog daemon, so listening to UDP port 514 is enough.

And surely, we got the little box to talk with us:

[syslog from 192.168.100.50:514] <15> 0x0.0x48b 0:14:4f:85:f2:b Network: soff 0 n 1
[syslog from 192.168.100.50:514] <15> 0x0.0x48b 0:14:4f:85:f2:b Network: GetNextAltAuth 0 0 0
[syslog from 192.168.100.50:514] <15> 0x0.0x48b 0:14:4f:85:f2:b Application: Stop tokens 8009df20
[syslog from 192.168.100.50:514] <15> 0x0.0x48b 0:14:4f:85:f2:b Application: sessionMonitor TcpOpen timeout to 0.0.0.0:7009
[syslog from 192.168.100.50:514] <15> 0x0.0x48b 0:14:4f:85:f2:b Network: soff 0 n 1
[syslog from 192.168.100.50:514] <15> 0x0.0x48b 0:14:4f:85:f2:b Network: GetNextAltAuth 0 0 0
[syslog from 192.168.100.50:514] <15> 0x0.0x873 0:14:4f:85:f2:b Application: getNextServer: no message - timeout

Gorgeous! It can talk, and the log levels do map cleanly to syslog severities. That <15> is a PRI value in the syslog protocol, obtained by (facility * 8) + severity, as defined by the spec. Working backwards:

15 / 8 = 1 remainder 7

Meaning:

• Facility: 1 (aka user-level messages)
• Severity: 7 (aka Debug)

The Authentication Server

It seems to be trying to do something, but failing. That’s expected, since the essential value is still missing from the Vendor TLV: AuthSrvr. It’s even marked as mandatory! One funny detail about that table is that, although the BarrierLevel is also marked as mandatory, the client continues the process normally. More on that later.

Now let’s set the AuthSrvr value. It’s the same as LogHost, but with a different tag:

• 0x15, 0x04, 0xC0, 0xA8, 0x64, 0x01. (AuthSrv, 4 bytes, 192.168.100.1)

After updating what’s mimicking the DHCP server the Ray expects and adding the potentially last flag it wants, we have new stuff on the syslog!

[syslog from 192.168.100.50:514] <15> 0x0.0x1e9 0:14:4f:85:f2:b Network: GetNextAltAuth 0 c0a86401 0
[syslog from 192.168.100.50:514] <15> 0x0.0x1e9 0:14:4f:85:f2:b Application: Stop tokens 8009df20
[syslog from 192.168.100.50:514] <15> 0x0.0x1e9 0:14:4f:85:f2:b Application: sessionMonitor TcpOpen timeout to 192.168.100.1:7009
[syslog from 192.168.100.50:514] <15> 0x0.0x1e9 0:14:4f:85:f2:b Network: GetNextAltAuth 0 c0a86401 0
[syslog from 192.168.100.50:514] <15> 0x0.0x1fd 0:14:4f:85:f2:b Network: GetNewtBandwidth: status down speed 100000000 mode full

There’s one little thing that may catch your eye: Application: sessionMonitor TcpOpen timeout to 192.168.100.1:7009. TCPOpen? To the IP of the AuthSrvr? Port 7009? That’s oddly familiar! We had a mysterious UDP packet going to that port in the last post!

The Mysterious serverQ

To be honest, serverQ was a dead-end. I couldn’t find any references on what it means, how it should respond, and why that happens. But hey! Now we have the Ray trying to open a full-fledged TCP connection with us! I mean, it’s trying, and that’s progress! I’m calling this out explicitly because the previous post ended on a bit of a cliffhanger, and I hate those as much as you probably do. For now, serverQ remains unexplained, but it no longer blocks progress.

Moving on!

A Sun Ray walks into a bar…

Since the Ray wants to speak, let it speak! Let’s spin a TCP server listening on port 7009 and dump whatever we get there. Here’s a few bytes of output from the hacky TCP server:

DummyAuthSrvr: 696e666f526571204d54553d31353...

For the well-trained eye, that hex looks an awful lot like an ASCII blob. And guess what? It is! Here’s the ASCII version of it:

infoReq MTU=1500 _=1 barrierLevel=310 cause=insert clientRand=nrlifRNCnn4xe.F5Be/DA7gkwFF9cOXQOmxxzu8SM3e ddcconfig=1 event=insert firstServer=c0a86401 fw=MfgPkg_4.15_2006.07.20.16.57 hw=SunRayP8 id=00144f85f20b initState=1 namespace=IEEE802 pn=53622 realIP=c0a86432 sn=00144f85f20b startRes=1400x1050:1400x1050 state=disconnected tokenSeq=1 type=pseudo

And that’s A LOT! Holy guacamole!

Let’s try to break it up:

• infoReq: Possibly an operation identifier?
• MTU: The MTU we already provided to it.
• _: No idea. Maybe a placeholder, maybe something deprecated. Who knows!
• barrierLevel: Possibly the BarrierLevel we didn’t provide it. Something related to firmware download, but let’s leave that for another time.
• cause: Perhaps what made the Ray request that operation?
• clientRand: A Base64! Sweet! Given the entropy, it’s really random! Perhaps for some kind of negotiation? I think we will eventually figure that out.
• ddcconfig: No idea.
• event: The same value of cause? We will need some other packages to figure this one out as well.
• firstServer: That’s the server’s IP in hexadecimal. Nothing interesting so far.
• fw: A firmware version! Seems like this one is from 2006. Cute!
• hw: A hardware identifier! It is a SunRayP8. Duly noted.
• id: I don’t know, but this value is static across all messages it sent. (And resent.)
• initState: Not a clue!
• namespace: IEEE802 is a fancy name for Ethernet. Maybe this thing (or other revisions) could support other connection methods?
• pn: This value changes in a non-monotonic fashion. Packet number? We will figure out.
• realIP: This is the Ray’s IP in hexadecimal.
• sn: Also static across all packets. Possibly a serial number.
• startRes: It’s initial resolution?
• state: Disconnected? Maybe because authentication hasn’t completed?
• tokenSeq: I’m sure we will eventually figure out as well.
• type: pseudo is an interesting choice. I’m interested, but not a clue.

Then, it immediately sends another packet:

DummyAuthSrvr: 6b656570416c697665526571205f3...

Again, more ASCII:

keepAliveReq _=1 fw=MfgPkg_4.15_2006.07.20.16.57 hw=SunRayP8 namespace=IEEE802 pn=53622 sn=00144f85f20b state=disconnected

And we have more details now!

1. Seems like we have another operation: keepAliveReq
2. pn is the same across both messages. Possibly not a packet number!
3. Everything else stays the same.

Since we’re not answering after that point it just eventually times out and retries:

[syslog from 192.168.100.50:514] <15> 0x0.0x16ca1 0:14:4f:85:f2:b Network: TcpReceive: FIN received 805bdd78 1
[syslog from 192.168.100.50:514] <15> 0x0.0x16ca1 0:14:4f:85:f2:b Application: restartAuth
[syslog from 192.168.100.50:514] <15> 0x0.0x16ca1 0:14:4f:85:f2:b Network: TcpClose: 805bdd78
[syslog from 192.168.100.50:514] <15> 0x0.0x16ca1 0:14:4f:85:f2:b Application: TcpClose1 failed
[syslog from 192.168.100.50:514] <14> 0x0.0x16ca1 0:14:4f:85:f2:b Network: FreeTcb:Freeing Tcb before the connection is closed

It’s not rebooting, though. Only restarting the authentication phase. It may be worth noticing that whenever it retries, even without rebooting both clientRand and pn changes, but everything else stays the same. There’s one small detail, though! Here’s a new detail in syslog:

[syslog from 192.168.100.50:514] <13> 0x0.0x544 0:14:4f:85:f2:b USB: rdd: unable to contact device manager on 192.168.100.1:7011  retrying

It’s ALSO trying to establish a connection to port 7011! Looking at my packet dumps, it tried to open a TCP connection to that port. Let’s also see what happens if we let it! It seems to be something called a Device Manager, so let’s do the same we did with the AuthSrvr:

DummyDeviceManager: 636f6e6e6563740a
[syslog from 192.168.100.50:514] <15> 0x0.0x1f270 0:14:4f:85:f2:b Network: TcpClose: 8059c538
[syslog from 192.168.100.50:514] <15> 0x0.0x1f270 0:14:4f:85:f2:b Network: TcpReceive: FIN received 8059c538 8
[syslog from 192.168.100.50:514] <14> 0x0.0x1f2d8 0:14:4f:85:f2:b Network: TCBDaemon: packet receive error
[syslog from 192.168.100.50:514] <14> 0x0.0x1f33c 0:14:4f:85:f2:b Network: TCBDaemon: packet receive error
[syslog from 192.168.100.50:514] <14> 0x0.0x1f3a0 0:14:4f:85:f2:b Network: TCBDaemon: packet receive error
...

Yeah, it’s not happy. Again, an ASCII payload. This time it only says connect, followed by a newline.

Eventually it will close the connection and send another connect, while the AuthSrvr continues receiving keepAliveReq.

For now, that’s it. I’ll eventually rewrite the C stuff in Golang just for the sake of it, and keep it updated in a Git repository. This way we can isolate the AuthSrvr, Device Manager, and the hacky DHCP server in a single place.

I said I hate cliffhangers, and I meant it, but unfortunately the Sun Ray just gave me a really good one.

In the next post, we’ll respond to its requests and see what happens when we get it wrong.

The Ray is a thin client that connects to a remote Sun server. I already ordered a Sun Fire, but until it arrives, I decided to poke the little thing and see what makes it tick!

After powering on, and having a network cable connected, the Ray shows a little screen with its MAC address, the network cable speed, and if it’s reached a local server. Since the Sun Fire is not here, the last part just does not happen, so let’s begin by observing how it tries to reach the server.

DHCPDISCOVER

The Ray starts with a DHCPDISCOVER to try to reach its server, which is no big surprise:

0000   ff ff ff ff ff ff 00 14 4f 85 f2 0b 08 00 45 00   ........O.....E.
0010   01 48 00 28 40 00 ff 11 7a 7d 00 00 00 00 ff ff   .H.(@...z}......
0020   ff ff 00 44 00 43 01 34 00 00 01 01 06 00 ff 4e   ...D.C.4.......N
0030   c3 a7 04 7e 00 00 00 00 00 00 00 00 00 00 00 00   ...~............
0040   00 00 00 00 00 00 00 14 4f 85 f2 0b 00 00 00 00   ........O.......
0050   00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   ................
0060   00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   ................
0070   00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   ................
0080   00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   ................
0090   00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   ................
00a0   00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   ................
00b0   00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   ................
00c0   00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   ................
00d0   00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   ................
00e0   00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   ................
00f0   00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   ................
0100   00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   ................
0110   00 00 00 00 00 00 63 82 53 63 35 01 01 3c 0e 53   ......c.Sc5..<.S
0120   55 4e 57 2e 4e 65 77 54 2e 53 55 4e 57 3d 07 01   UNW.NewT.SUNW=..
0130   00 14 4f 85 f2 0b ff 00 00 00 00 00 00 00 00 00   ..O.............
0140   00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   ................
0150   00 00 00 00 00 00                                 ......

DIX / Ethernet II

The first portion is a classic DIX Ethernet frame (or Ethernet II, if you’re still litigating DIX credit). From 0x00 to 0x05 we have the broadcast address ff:ff:ff:ff:ff:ff, the second one is the Ray’s address, 00:14:4f:85:f2:0b, 0x0C to 0x0D indicates an IPv4 packet (0x0800).

IPv4

The second portion, as expected, is the IPv4 header. Nothing new there as well, the only relevant information there is that the destination address is 255.255.255.255, and it’s a UDP packet; this basically covers bytes 0x0E to 0x21.

UDP

From 0x22 to 0x29 we can observe the UDP header. Interesting points are source port 68, and destination port 67.

BOOTP/DHCP fixed header + DHCP options

From 0x2A onwards we have the DHCPDISCOVER payload. Finally something interesting:

• Message Type is 0x01, meaning it’s a Boot Request,
• Hardware Type is 0x01, or “Ethernet”
• Hardware Address Length is 0x06
• Hops is set as 0x00
• There’s a Transaction ID (0xff4ec3a7)
• A “Seconds Elapsed” field, from the BOOTP era (which the Ray is actively using!). This is an interesting find: nowadays many implementations don’t do much with this field, and plenty of clients just leave it at zero, but this was important back then for a few different reasons! This field aided:
  • Relay agents decide whether to forward a request
  • Servers decide whether to prioritize a client that has been waiting longer
  • Debugging slow or failing DHCP negotiations
• BOOTP flags all zeroed; basically indicating the client is capable of receiving unicast replies
• Zeroed IP addresses
• The Ray’s MAC address, again
• A lot of padding for the Ray’s hardware address
• An empty server hostname
• An empty boot file name
• The Magic Cookie
• And FINALLY what I’m looking for:
  • Option 53, indicating the discovery packet
  • Option 60, containing the vendor class identifier
    • The value is SUNW.NewT.SUNW
  • Option 61 identifies the client using a hardware-type-prefixed identifier (Ethernet + MAC address), effectively restating the Ray’s MAC in a DHCP-standardized form. Again.
  • Option 255, indicating no more options follow
• And a lot of padding.

So what’s interesting so far?

The big oddity is what’s missing: Option 55 (Parameter Request List). Most DHCP clients use it to ask for things like DNS servers, routes, NTP, etc. The Sun Ray doesn’t ask, it just identifies itself (Vendor Class + Client ID) and waits to see what comes back. That suggests this client either runs with strong defaults, or expects a Sun-aware environment to “just know” what to provide.

Also: SUNW.NewT.SUNW looks like a Sun vendor-class marker; possibly a New Terminal generation marker rather than a generic Sun Ray family identifier.

This is the first part of the whole negotiation, and lots can already be observed and speculated!

Now, let’s make something that replies the Ray as it expects. I could just leverage dnsmasq for that, but let’s go the fun way(tm) and write a bit of C:

//
// Created by vitosartori on 12/13/25.
//

// Minimal DHCP responder (DISCOVER->OFFER, REQUEST->ACK) for RE labs.
// Build:  gcc -O2 -Wall -Wextra -o ray_dhcp ray_dhcp.c
// Run:    sudo ./ray_dhcp enp2s0 192.168.100.1 192.168.100.50 255.255.255.0 192.168.100.1

#define _GNU_SOURCE
#include <arpa/inet.h>
#include <errno.h>
#include <netinet/in.h>
#include <stdint.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <sys/socket.h>
#include <sys/types.h>

enum {
    BOOTREQUEST = 1,
    BOOTREPLY   = 2,

    HTYPE_ETHERNET = 1,
    HLEN_ETHERNET  = 6,

    DHCP_COOKIE = 0x63825363,

    // DHCP Options

    OPT_PAD        = 0,
    OPT_SUBNETMASK = 1,
    OPT_ROUTER     = 3,
    OPT_REQ_IP     = 50,
    OPT_LEASETIME  = 51,
    OPT_MSGTYPE    = 53,
    OPT_SERVERID   = 54,
    OPT_VENDORCLASS= 60,
    OPT_CLIENTID   = 61,
    OPT_END        = 255,

    // DHCP Message Types

    DHCPDISCOVER = 1,
    DHCPOFFER    = 2,
    DHCPREQUEST  = 3,
    DHCPACK      = 5
};

#pragma pack(push, 1)
typedef struct {
    uint8_t  op;
    uint8_t  htype;
    uint8_t  hlen;
    uint8_t  hops;
    uint32_t xid;
    uint16_t secs;
    uint16_t flags;
    uint32_t ciaddr;
    uint32_t yiaddr;
    uint32_t siaddr;
    uint32_t giaddr;
    uint8_t  chaddr[16];
    uint8_t  sname[64];
    uint8_t  file[128];
    // Following is cookie + TLVs
} bootp_t;
#pragma pack(pop)

typedef struct {
    uint8_t  msgtype;       // Option 53
    uint8_t  has_msgtype;
    uint8_t  vendor[256];   // Option 60
    uint8_t  vendor_len;
    uint8_t  has_vendor;
    uint32_t req_ip;        // Option 50 (optional)
    uint8_t  has_req_ip;
} dhcp_opts_t;

static void die(const char *msg) {
    perror(msg);
    exit(1);
}

static int parse_ipv4(const char *s, uint32_t *out_nbo) {
    struct in_addr a;
    if (inet_pton(AF_INET, s, &a) != 1) return 0;
    *out_nbo = a.s_addr;
    return 1;
}

static void print_mac(const uint8_t mac[6], char *buf, const size_t buf_len) {
    snprintf(buf, buf_len, "%02x:%02x:%02x:%02x:%02x:%02x",
             mac[0], mac[1], mac[2], mac[3], mac[4], mac[5]);
}

static int parse_dhcp_options(const uint8_t *p, const size_t n, dhcp_opts_t *o) {
    memset(o, 0, sizeof(*o));

    // First the cookie
    if (n < 4) return 0;
    uint32_t cookie;
    memcpy(&cookie, p, 4);
    if (ntohl(cookie) != DHCP_COOKIE) {
        fprintf(stderr, "drop: cookie mismatch (got %08x)\n", ntohl(cookie));
        return 0;
    }

    size_t i = 4;

    while (i < n) {
        const uint8_t code = p[i++];
        if (code == OPT_PAD) continue;
        if (code == OPT_END) break;
        if (i >= n) break;

        const uint8_t len = p[i++];
        if (i + len > n) break;
        const uint8_t *val = p + i;

        if (code == OPT_MSGTYPE && len == 1) {
            o->msgtype = val[0];
            o->has_msgtype = 1;
        } else if (code == OPT_VENDORCLASS && len > 0 && len <= 255) {
            memcpy(o->vendor, val, len);
            o->vendor[len] = 0;
            o->vendor_len = len;
            o->has_vendor = 1;
        } else if (code == OPT_REQ_IP && len == 4) {
            memcpy(&o->req_ip, val, 4);
            o->has_req_ip = 1;
        }

        i += len;
    }

    return 1;
}

static uint8_t *opt_put(uint8_t *w, const uint8_t code, const void *val, const uint8_t len) {
    *w++ = code;
    *w++ = len;
    memcpy(w, val, len);
    return w + len;
}

static size_t build_reply(uint8_t *out,
                          const size_t outcap,
                          const bootp_t *req,
                          const dhcp_opts_t *reqo,
                          const uint8_t dhcp_type,
                          const uint32_t server_ip,
                          const uint32_t yiaddr,
                          const uint32_t subnet_mask,
                          const uint32_t router_ip,
                          const uint32_t lease_seconds) {
    if (outcap < sizeof(bootp_t) + 4 + 64) return 0;

    bootp_t rep = {0};

    rep.op = BOOTREPLY;
    rep.htype = req->htype;
    rep.hlen = req->hlen;
    rep.hops = 0;
    rep.xid = req->xid;
    rep.secs = 0;
    rep.flags = req->flags; // Echo client flags

    rep.ciaddr = 0;
    rep.yiaddr = yiaddr; // Offered/acked IP
    rep.siaddr = server_ip; // the mothership hint
    rep.giaddr = 0;

    memcpy(rep.chaddr, req->chaddr, 16);

    memcpy(out, &rep, sizeof(rep));
    uint8_t *w = out + sizeof(rep);

    const uint32_t cookie = htonl(DHCP_COOKIE);
    memcpy(w, &cookie, 4);
    w += 4;

    // Options
    w = opt_put(w, OPT_MSGTYPE, &dhcp_type, 1);
    w = opt_put(w, OPT_SERVERID, &server_ip, 4);

    const uint32_t lt = htonl(lease_seconds);
    w = opt_put(w, OPT_LEASETIME, &lt, 4);

    w = opt_put(w, OPT_SUBNETMASK, &subnet_mask, 4);
    w = opt_put(w, OPT_ROUTER, &router_ip, 4);

    // Optional: echo vendor class back, may help with “vendor-aware feel”
    if (reqo->has_vendor && reqo->vendor_len > 0) {
        w = opt_put(w, OPT_VENDORCLASS, reqo->vendor, reqo->vendor_len);
    }

    *w++ = OPT_END;

    return (size_t) (w - out);
}

int main(const int argc, char **argv) {
    if (argc != 6) {
        fprintf(stderr,
            "Usage: %s <ifname> <server_ip> <offer_ip> <subnet_mask> <router_ip>\n"
            "Example: %s enp2s0 192.168.100.1 192.168.100.50 255.255.255.0 192.168.100.1\n",
            argv[0], argv[0]);
        return 2;
    }

    const char *ifname = argv[1];
    uint32_t server_ip, offer_ip, mask_ip, router_ip;

    if (!parse_ipv4(argv[2], &server_ip) ||
        !parse_ipv4(argv[3], &offer_ip)  ||
        !parse_ipv4(argv[4], &mask_ip)   ||
        !parse_ipv4(argv[5], &router_ip)) {
        fprintf(stderr, "Invalid IPv4 argument\n");
        return 2;
    }

    const int s = socket(AF_INET, SOCK_DGRAM, 0);
    if (s < 0) die("socket");

    const int yes = 1;
    if (setsockopt(s, SOL_SOCKET, SO_REUSEADDR, &yes, sizeof(yes)) < 0) die("setsockopt REUSEADDR");
    if (setsockopt(s, SOL_SOCKET, SO_BROADCAST, &yes, sizeof(yes)) < 0) die("setsockopt BROADCAST");

    // Bind to port 67 on all addresses
    struct sockaddr_in bindaddr = {0};
    bindaddr.sin_family = AF_INET;
    bindaddr.sin_port = htons(67);
    bindaddr.sin_addr.s_addr = htonl(INADDR_ANY);

    if (bind(s, (struct sockaddr*)&bindaddr, sizeof(bindaddr)) < 0) {
        fprintf(stderr, "bind(:67) failed: %s\n", strerror(errno));
        fprintf(stderr, "Tip: run as root or grant cap_net_bind_service.\n");
        return 1;
    }

    // Pin to interface so we don't respond on the wrong NIC.
    if (setsockopt(s, SOL_SOCKET, SO_BINDTODEVICE, ifname, strlen(ifname) + 1) < 0) {
        fprintf(stderr, "SO_BINDTODEVICE(%s) failed: %s\n", ifname, strerror(errno));
        return 1;
    }

    fprintf(stderr, "Listening on %s UDP/67. server=%s offer=%s\n", ifname, argv[2], argv[3]);

    uint8_t inbuf[2048];
    uint8_t outbuf[2048];

    for (;;) {
        struct sockaddr_in src;
        socklen_t srclen = sizeof(src);
        const ssize_t n = recvfrom(s, inbuf, sizeof(inbuf), 0, (struct sockaddr*)&src, &srclen);
        if (n < 0) {
            if (errno == EINTR) continue;
            die("recvfrom");
        }
        fprintf(stderr, "got %zd bytes from %s:%d\n",
        n, inet_ntoa(src.sin_addr), ntohs(src.sin_port));
        fflush(stderr);
        enum { BOOTP_FIXED_LEN = 236 };

        if ((size_t)n < BOOTP_FIXED_LEN + 4) continue;
        const size_t opt_len = (size_t)n - BOOTP_FIXED_LEN;

        const bootp_t *req = (const bootp_t *)inbuf; // prolly OK for chaddr/xid/etc.
        if (req->op != BOOTREQUEST) { fprintf(stderr,"drop: not BOOTREQUEST\n"); continue; }
        if (req->htype != HTYPE_ETHERNET) { fprintf(stderr,"drop: htype=%u\n", req->htype); continue; }
        if (req->hlen != HLEN_ETHERNET) { fprintf(stderr,"drop: hlen=%u\n", req->hlen); continue; }

        // Options start at offset 236
        dhcp_opts_t o = {0};
        if (!parse_dhcp_options(inbuf + BOOTP_FIXED_LEN, opt_len, &o)) continue;
        if (!o.has_msgtype) {
            fprintf(stderr,"drop: no option 53\n");
            continue;
        }

        char macs[32];
        print_mac(req->chaddr, macs, sizeof(macs));

        fprintf(stderr, "opts[0..15]: ");
        for (int i = 0; i < 16 && (size_t)i < opt_len; i++) fprintf(stderr, "%02x ", inbuf[BOOTP_FIXED_LEN + i]);
        fprintf(stderr, "\n");

        uint8_t reply_type = 0;
        if (o.msgtype == DHCPDISCOVER) reply_type = DHCPOFFER;
        else if (o.msgtype == DHCPREQUEST) reply_type = DHCPACK;
        else {
            fprintf(stderr, "drop: Unknown msgtype 0x%02x\n", o.msgtype);
            continue;
        }

        // Honor Option 50 Requested IP if present.
        uint32_t yiaddr = offer_ip;
        if (o.has_req_ip) yiaddr = o.req_ip;

        size_t outlen = build_reply(outbuf, sizeof(outbuf), req, &o, reply_type,
                                    server_ip, yiaddr, mask_ip, router_ip, 3600);
        if (!outlen) continue;

        // Send to broadcast:68, as we noticed the client doesn't have an IP
        struct sockaddr_in dst = {0};
        dst.sin_family = AF_INET;
        dst.sin_port = htons(68);
        dst.sin_addr.s_addr = htonl(INADDR_BROADCAST);

        if (sendto(s, outbuf, outlen, 0, (struct sockaddr*)&dst, sizeof(dst)) < 0) {
            fprintf(stderr, "sendto failed: %s\n", strerror(errno));
            continue;
        }

        fprintf(stderr, "Sent %lu bytes\n", outlen);

        if (o.has_vendor) {
            fprintf(stderr, "%s xid=%08x -> %s (vendor=%s)\n",
                    macs, ntohl(req->xid),
                    (reply_type == DHCPOFFER) ? "OFFER" : "ACK",
                    (const char *)o.vendor);
        } else {
            fprintf(stderr, "%s xid=%08x -> %s\n",
                    macs, ntohl(req->xid),
                    (reply_type == DHCPOFFER) ? "OFFER" : "ACK");
        }
    }
}

This replies with what the Ray expects, offering it an IP leased from the Linux box. I only forgot to mention how things are wired here at my lab:

Sun Ray ↔ USB Ethernet ↔ Linux box (DHCP responder) ↔ rest of LAN

So the magic is happening on interface enx00e04c3f6690, I’ll give myself an IP 192.168.100.1, and give 192.168.100.50 to the Ray. First, let’s get an IP on that interface:

sudo ip addr add 192.168.100.1/24 dev enx00e04c3f6690
sudo ip link set enx00e04c3f6690 up

And finally compile and run (as sudo). The result is positive! The Ray updated the UI indicating it has communicated with the server! Then, a wild UDP packet appears:

0000   ff ff ff ff ff ff 00 14 4f 85 f2 0b 08 00 45 00   ........O.....E.
0010   00 27 00 06 40 00 40 11 15 e6 c0 a8 64 32 ff ff   .'..@.@.....d2..
0020   ff ff 1b 61 1b 61 00 13 00 00 73 65 72 76 65 72   ...a.a....server
0030   51 3d 31 0a 00 00 00 00 00 00 00 00               Q=1.........

Ignoring the framing information and other headers we get 11 payload bytes plus some padding. The payload is:

0000 73 65 72 76 65 72 51 3d 31 0a 00                   serverQ=1..

That’s… Literally serverQ=1\n\0. But there’s more! Here’s what the packet gives:

1. Both source and destination ports are 7009
2. The source IP is the IP we gave to the Ray! (I omitted a few DHCPACK and DHCPINFORM packets)
3. The destination IP is STILL the broadcast address

This might be Sun Ray’s appliance protocol territory (UDP/7009 shows up in Sun Ray land), but I’m treating it as an unknown for now. The protocol is proprietary and not documented, but let’s wrap up for now and just be happy for the little, still important, progress.

Next: what happens if we answer serverQ=1?

1. I despise LLMs. I don’t think they should replace or even try to replace people.
2. I am a big fan of well-written RFCs and specifications.

The first point is not a surprise; being a part of this industry before the advent of LLMs gave me enough resources to be able to compare how things used to work, and how they work now. The field is clearly divided, though, but I see it as as binary as ever: either engineers love it, or despise it. I feel that people are still in awe of the possibility to have something else capable of, for instance, to write unit tests for a feature, but they fail to understand that the one who wrote the implementation must also be the one who writes the tests, as albeit manual, the process is still important, since it forces one to use what they implemented. In this step, several extremely relevant points can still be observed:

• The ergonomics of the API. Is it cohesive? How hard is it for someone to use it?
• Its documentation. Is it documented at all? Will someone that has absolutely no context of the whole be able to still understand it and use without friction?
• Is the domain properly segregated? Is the API requiring more than it should?
• Are names descriptive enough? Do they make sense when used with other components?

Those are just a glimpse of what writing both tests and documentation can provide to the one who is implementing it, but the list goes on. Now, delegating the job to a tool whose reasoning is not even visible blinds implementors from those nuances.

I also see how this is deeply affecting those who are entering the field. For instance, when mentoring anyone, the most important thing I keep repeating on and on is: Read and understand error messages, and do not ignore warnings. I see often people just acknowledging message boxes on UIs, hitting OK without even reading it, and then asking themselves “Why isn’t this working?”; the same applies to developers. One must be able to understand what an error message means, and how to read a stack trace. Now, imagine someone in university, or in an internship blindly writing code, and asking an LLM to interpret any errors and fix them. What will this person be able to accomplish without it?

But this is not only affecting newcomers. I often see interesting cases coming from people who are part of the field for a while:

• “Your code is wrong, see what said: ..."; meaning "I haven't read your code, I have no idea what it does, and I'm acting as a proxy to an LLM".
• “I think we should use , see my chat transcription": meaning "I have no domain nor property, but you should use what the machine told me"
• I opened a Pull Request, but I have no idea what it does, as seen in the infamous Mesa Issue.
• “There’s something wrong, see what I got regarding this issue using ".

Sometimes I feel like people can’t even read and maintain their own code, let alone code generated by an LLM, copy-and-pasted into the codebase; I don’t need to say that’s dangerous not only on a security point-of-view, but on a business continuation one.

This ties directly into my second point about well-written specifications, which exist precisely to prevent this kind of ambiguity and dependency on individual authors.

The second point is also not a surprise. I’m a big fan of LaTeX, and a bigger fan of IETF and everything they achieved. I work with the premise that every RFC and specification must be written in a fashion that allows anyone reading it to fully understand what it is about, and even more important: to be able to implement it no matter the language or technology they want to use. This way, even if I leave the company I’m working on, the documentation is there, all the design decisions are still there, and are both navigable and clear to the point where they don’t need me, the original author, to be able to update or reimplement it: everything is within a single documentation unit.

Now, where can LLMs help us here? Not designing documentation, not writing code, but reviewing the documents. It turns out that LLMs are great reviewers, since they can keep a lot in their context, and are also able to comprehend documents in a matter of seconds. However, there’s an important pitfall in leveraging hosted LLMs (like ChatGPT or Claude): one must be absolutely sure of their data policies, and even more important, that the company will not use anything you provided to train future versions. So that is how I use LLMs: for reviewing what I write.

Given their ability to quickly cross-examine a whole document in seconds, I mostly use it as part of my writing process. First, I draft the document, put everything I need, and make the first review. When I’m happy with the result, it’s time to get a critic. I allow the LLM to read it, and ask for feedback on sections that need development, for points that are not clear or ambiguous, for contradictions and typos. And they are excellent at this. They can pinpoint which section makes a contradictory statement against another, potentially incomplete definitions, examples or small details that are present in an ABNF section, but explained incorrectly in the body of text, along other small nits and inconsistencies that can truly elevate the level of the document. But do notice how it didn’t replace the creation process, it is not replacing the brain, only augmenting it to the point where one would forget that a small sentence ten sections earlier didn’t match what’s written in the end of the document.

That’s what people should be using LLMs for. For augmenting our abilities. Not replacing them. Renata Martins wrote a few months ago a post with a provocative title: LLMs are the Lobotomy of the 21st Century, and time and time again I fear that it may be true.

This post diverged from what I’m used to writing, but I had this on my head for so long, that I thought it was worth putting out.

For now, let’s continue using our brains as we should, and avoid atrophying them; if we use LLMs thoughtfully, they can strengthen (rather than weaken) our engineering practices.

According to the git-commit manpage, it’s “a good idea to begin the commit message with a single short (no more than 50 characters)”. This commit message lands at 71 characters. Personally, I would have put something like “Fix array parsing when input has multiple spaces”. Smaller, and also complies with recommendations from the git documentation.

The type of a commit here is only really useful for automated tools, usually the type can be inferred from the summary of the commit. As such, I think the type would be better off as a field in footer of commit, under a key such as Commit-Type or Type.

But why make an exception for BREAKING CHANGE ? Why not have it as Breaking-Change ? It’s not like it adds or remove any character, all that would change is it would stay consistent with other keys.

That makes so much sense. And folks, we’re just a --trailer away from making this work as it should.

Now for the facts: I do use conventional commits, as it is — as Sarah said — useful for automated tools, and every time I make a commit message larger than it should be, I do feel bad. However, adding --trailer for each commit and for each property would be cumbersome. So why not make a wrapper for git commit to handle that?

The Idea

What if we could do something like git commit --chore --breaking -m "Message here" and get those fancy trailers? Of course, we have some commits like chore(some-component): Some description, so let’s also allow --chore to accept an optional parameter. I’m low on imagination today, so let’s call this “Pretty Commit”, and alias it as pc. Here are some possibilities:

git pc --chore component -m "Commit message"
# or
git pc --spec -m "Commit message"
# or even
git pc --feat "Tokenizer" --breaking -m "Allow tokenizer to do a backflip"

And the result from the last commit:

commit 4decde3d186a834a092676d3143fae6f7a54945a (HEAD -> master)
Author: Vito Sartori <hey@vito.io>
Date:   Mon Jul 7 10:50:22 2025 -0300

    Allow tokenizer to do a backflip

    Type: feat
    Scope: Tokenizer
    Breaking: true

All parseable! Let’s, for instance, get the meta from this commit:

 λ | git log -1 --pretty=format:"%B" | git interpret-trailers --parse
Type: feat
Scope: Tokenizer
Breaking: true

That’s so much easier to parse than extracting stuff from the commit message alone. Git plumbing has everything everyone ever needed.

I’ll try using this approach and also use it as a reason to bump heyvito/semver-releaser.

Can I use it?

Of course! Git is composable, meaning you just need to have git-pc in your PATH. Then git pc is immediately available. The source is available at github.com/heyvito/git-pc, and is a (very) simple bash script — as far as ‘simple’ and ‘bash’ can go together.

Contributions are welcome!

What else?

If you didn’t, read Sarah’s post. It’s full of insights and was what made me do this. Also, Tyler Cipriani posted about Git Notes, which is another feature that goes unnoticed.

And that’s it for today. Let’s see if we can push this forward!

Now it’s been a while since The Browser Company released a compelling update that’s not just a bump to the internal Chromium version, and filling their release notes with irrelevant content. Of course, they are focusing on their AI-based browser Dia, but I feel betrayed, considering all the promises they made.

There’s one thing that kept me on Arc, though: its sidebar. The sidebar was a great concept, and I could simply drag-and-drop tabs to read later, and it would just sync across my computers. That was great — until it became a mess. My sidebar is long. I mean really long. And searching/exploring it became hard. Adding all that up, I felt it was time to switch back to Firefox — and switch back I did.

But, what about my sidebar? Well, Arc does not offer a way to export the sidebar, but luckily, it keeps it in a JSON file in its Application Support directory. So a quick Ruby script later, all my links were exported. The point then was: where to place those items?.

Fixing my Bookmarks Issue

The first place I looked was raindrop.io, which I used for a while after it launched. Raindrop is great — really — but it has one small issue that I can’t stand: It does not permanently store YouTube videos — which makes sense, I think.

So I started working on my own solution to that: Rio (Retain Information Onshore). Rio takes a URL or file from my system, processes it, and permanently stores it, be it a web page, PDF, docx, or video. With a little help from Llama/Mistral, PDFs are correctly summarized and have their title extracted, so everything stays organized.

I have plans to also add a Solr integration for full-text search, but that will have to wait longer.

This is somewhat a big project, and may be interesting to other people who like to self-host stuff, so eventually it may become open-source. Who knows :)

So, so long Arc. It was good while you lasted. Let’s rock Firefox Nightly and keep the web balanced.

First Request: No If-None-Match

GET /react HTTP/1.1
Host: registry.npmjs.org
Connection: close
User-Agent: [REDACTED]/0.1.0

The response is expected:

HTTP/1.1 200 OK
Date: Thu, 30 Jan 2025 12:35:30 GMT
Content-Type: application/json
Content-Length: 5387821
Connection: close
CF-Ray: 90a18092cb3802e5-GRU
CF-Cache-Status: HIT
Accept-Ranges: bytes
Access-Control-Allow-Origin: *
Age: 75
Cache-Control: public, max-age=300
ETag: "5611f329debbb41fda058497d0d4c7d8"
Last-Modified: Wed, 29 Jan 2025 16:20:20 GMT
Vary: accept-encoding, accept
Server: cloudflare

Fancy stuff. Very nice. Even an ETag! So the expectation is that If-None-Match yields an HTTP 304, right? Let’s try it again:

GET /react HTTP/1.1
If-None-Match: "5611f329debbb41fda058497d0d4c7d8"
Host: registry.npmjs.org
Connection: close
User-Agent: [REDACTED]/0.1.0

And…

HTTP/1.1 200 OK
Date: Thu, 30 Jan 2025 12:37:37 GMT
Content-Type: application/json
Content-Length: 5387821
Connection: close
CF-Ray: 90a183af0d59ae90-GRU
CF-Cache-Status: HIT
Accept-Ranges: bytes
Access-Control-Allow-Origin: *
Age: 202
Cache-Control: public, max-age=300
ETag: "5611f329debbb41fda058497d0d4c7d8"
Last-Modified: Wed, 29 Jan 2025 16:20:20 GMT
Vary: accept-encoding, accept
Server: cloudflare

OK? My If-None-Match is extactly the same ETag returned on the response.

Second Request: Should HEAD help us?

Okay, but WHAT IF we issued a HEAD in order to compare ETags? That would be enough to assert we have the same payload, right?

HEAD /react HTTP/1.1
Host: registry.npmjs.org
Connection: close
User-Agent: [REDACTED]/0.1.0

HTTP/1.1 200 OK
Date: Thu, 30 Jan 2025 12:39:41 GMT
Content-Type: application/json
Connection: close
CF-Ray: 90a186b3687af23f-GRU
CF-Cache-Status: HIT
Accept-Ranges: bytes
Access-Control-Allow-Origin: *
Age: 24
Cache-Control: public, max-age=300
ETag: W/"5611f329debbb41fda058497d0d4c7d8"
Last-Modified: Wed, 29 Jan 2025 16:20:20 GMT
Vary: accept-encoding, accept
Server: cloudflare

Wha-What? A Weak ETag? How? Why?

How NPM CLI handles this

After some digging, it seems NPM’s CLI is quite… simple. It internally caches the contents for 5 minutes. After that, it just downloads it again.

Workarounds

We could (which does not mean we should) read the initial headers and drop the connection, but that does not mean we won’t receive the payload on a GET request, we will only be breaking HTTP semantics. After all, the remote will be shoving the payload to our connection nonetheless.

Another option would be perform the same GET, but using Range to not get a body, as it seems the server supports it. Let’s try:

Leveraging Range Header

The server seems to support byte ranges, as announced by the header Accept-Ranges: bytes. So let’s ask for a single byte. We will waste a single byte, but at least it’s not 30-50MB depending on the manifest. I’ll keep the If-None-Match just for extra measure.

GET /react HTTP/1.1
If-None-Match: "5611f329debbb41fda058497d0d4c7d8"
Range: bytes=0-0
Host: registry.npmjs.org
Connection: close
User-Agent: [REDACTED]/0.1.0

HTTP/1.1 206 Partial Content
Date: Thu, 30 Jan 2025 12:54:36 GMT
Content-Type: application/json
Content-Length: 1
Connection: close
Content-Range: bytes 0-0/5387821
CF-Ray: 90a19c9078a51b24-GRU
CF-Cache-Status: HIT
Access-Control-Allow-Origin: *
Age: 11
Cache-Control: public, max-age=300
ETag: "5611f329debbb41fda058497d0d4c7d8"
Last-Modified: Wed, 29 Jan 2025 16:20:20 GMT
Vary: accept-encoding, accept
Set-Cookie: _cfuvid=hY7zxXExF2P4mja1cX6ynhOfcoAuJzxR42SmGTiLJvY-1738241676925-0.0.1.1-604800000; path=/; domain=.npmjs.org; HttpOnly; Secure; SameSite=None
Server: cloudflare

Success! We got a single byte, and a 206 status response. Is that enough? I honestly don’t think so, considering that the server returns an ETag. But if that’s the only alternative, I’ll happily stick with it, although it feels a really, really hacky solution.

Contacting NPM Support

I raised a ticket with NPM support to understand what is going on, but so far, I haven’t found a suitable alternative.

Will update this once I get a response from the fine folks at NPM.

One month later, I got the following reply from NPM Support:

So sorry for the oversight and delay addressing your concerns.

As previously mentioned, with the behavior you're seeing with the If-None-Match
header for registry.npmjs.org, it’s not expected for it to always send the full
payload when the ETag matches. Normally, if the If-None-Match header matches the
ETag returned by the registry, the server should respond with a 304 Not Modified
status, not the full payload.

It might be worth checking if there's anything else in the request (such as
additional headers or network issues) affecting the behavior.

As we’ve reached the maximum allowable time for keeping this ticket open, we
will be closing it. However, the issue is documented and escalated it to our
engineering team for further resolution.

We will continue working on this and encourage you to check periodically for any
updates on the status. I'll reach out with any updates.

We truly appreciate understanding throughout this process.

So… Not fixed, and perhaps won’t be. Bummer.

Here we will be configuring a cluster composed by:

• 1 Master Node
• 4 Worker Nodes
• 1 NFS server so we can use storage over network

I’m provisioning my cluster on ESXi that’s running behind a PFSense, so traffic can be controlled, and DHCP does not touch my main home network. I’ll also use an internal DNS to route everything, and will use domains *.k.vito.sh for exposing services hosted in the cluster, and *.kube.vito.sh to access VMS.

All machines will be provisioned with 8GB RAM and 4 cores. The network will look like this:

• Machine: k8s-master
  • IP: 10.0.11.0
  • DNS: master.kube.vito.sh
• Machine: k8s-worker1
  • IP: 10.0.11.1
  • DNS: w1.kube.vito.sh
• Machine: k8s-worker2
  • IP: 10.0.11.2
  • DNS: w2.kube.vito.sh
• Machine: k8s-worker3
  • IP: 10.0.11.3
  • DNS: w3.kube.vito.sh
• Machine: k8s-worker4
  • IP: 10.0.11.4
  • DNS: w4.kube.vito.sh
• Machine: k8s-nfs
  • IP: 10.0.11.5
  • DNS: nfs.kube.vito.sh

Provisioning Nodes

Warning: You will notice I’m running several commands without sudo. I usually just get a root shell using sudo su in order to execute everything without hassle.

All nodes except k8s-nfs follow the same steps:

1. Install Ubuntu 22.04 LTS. I use a minified server installation.
2. Make sure everything is up-to-date:
   1. apt update && apt upgrade -y && reboot
3. Drop snap
   1. apt purge snapd
4. Install utilities:
   1. apt install iputils-ping vim apt-transport-https ca-certificates curl gnupg2 gpg software-properties-common -y

Install Cri-O

1. export OS=xUbuntu_22.04
2. export CRIO_VERSION=1.26
3. echo "deb https://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable/$OS/ /"| sudo tee /etc/apt/sources.list.d/devel:kubic:libcontainers:stable.list
4. echo "deb http://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable:/cri-o:/$CRIO_VERSION/$OS/ /"|sudo tee /etc/apt/sources.list.d/devel:kubic:libcontainers:stable:cri-o:$CRIO_VERSION.list
5. curl -L https://download.opensuse.org/repositories/devel:kubic:libcontainers:stable:cri-o:$CRIO_VERSION/$OS/Release.key | sudo apt-key add -
6. curl -L https://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable/$OS/Release.key | sudo apt-key add -
7. apt update && apt install cri-o cri-o-runc -y
8. systemctl enable --now crio
9. Check Cri-O status: systemctl status crio

Enable conmon

1. Edit /etc/crio/crio.conf, find and uncomment the line conmon = ""
2. Insert /usr/bin/conmon between the quotes
3. Restart Cri-O: systemctl restart crio
4. Check its status: systemctl status crio

Install Cri-O tools

1. apt install -y cri-tools
2. Check everything is working: crictl --runtime-endpoint unix:///var/run/crio/crio.sock version
3. Check if it is ready: crictl info. It is expected RuntimeReady to be true

Configure Networking

This is optional, but since I use an internal DNS, I need to set nameservers through netplan:

1. Edit /etc/netplan/00-installer-config.yaml and add the following line: nameservers.addresses: [10.0.1.3]
2. Run netplan apply

Prepare APT for K8s installation

1. Make sure /etc/apt/keyrings exist. Create it if it does not:
   1. [ -d /etc/apt/keyrings ] || mkdir -p -m 755 /etc/apt/keyrings
2. Download public signing key for the K8s packages: curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.32/deb/Release.key | sudo gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg
3. Add the appropriate apt repository: echo 'deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.32/deb/ /' | sudo tee /etc/apt/sources.list.d/kubernetes.list
4. Install Kubernetes components:
   1. apt update
   2. apt install -y kubelet kubeadm kubectl
   3. apt-mark hold kubelet kubeadm kubectl
5. Enable and start kubelet: sudo systemctl enable --now kubelet

Update System Configuration for Kubernetes

1. Enable required kernel modules:
   1. modprobe overlay
   2. modprobe br_netfilter
2. Ensure modules are loaded on restart:

echo overlay | tee -a /etc/modules
echo br_netfilter | tee -a /etc/modules

1. Update network configuration:

cat > /etc/sysctl.d/99-kubernetes-cri.conf <<EOF
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward = 1
net.bridge.bridge-nf-call-ip6tables = 1
EOF

1. Apply system configuration: sysctl --system
2. Disable swap
   1. systemctl mask swap.img.swap
   2. systemctl stop swap.img.swap
   3. swapoff -a
3. Update kubeadm configuration to use systemd as its cgroup driver:
   1. Create /etc/default/kubelet with the following contents:

KUBELET_EXTRA_ARGS=--cgroup-driver=systemd --runtime-request-timeout=5m

1. Apply changes once again
   1. systemctl daemon-reload
   2. systemctl restart kubelet

At this point, kubelet will be crashing, waiting for kubeadm instructions.

Bootstrapping the Cluster

The next step must be performed only on the master node. This will bootstrap the cluster. Other nodes will join in another step.

To bootstrap the cluster, use kubeadm:

kubeadm init --pod-network-cidr=10.244.0.0/16 --control-plane-endpoint=master.kube.vito.sh

The command may take a while and will output instructions on how to allow non-root access to kubectl and how to join nodes to the cluster. Take note of those instructions!

Exit the root shell, and go back to your non-root user. Then:

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

Finally, run kubectl get ns. If you get a response, congratulations, you have a working master node!

Setup Flannel

Flannel is a CNI plugin that allows networking to work on your cluster. More information can be found here.

Installation is straightforward, requiring a single command:

kubectl apply -f https://github.com/flannel-io/flannel/releases/latest/download/kube-flannel.yml

Then you can use watch kubectl get pods --all-namespaces and wait for Flannel to bootup.

Setup Workers

Now, SSH into each of the workers and using information emitted by kubeadm, join them to the cluster. The command will look like the following, and must be executed with root privileges:

kubeadm join master.kube.vito.sh:6443 --token qn2gjj.e6y272oiqhqa2emu \
    --discovery-token-ca-cert-hash sha256:401e68dbd991b30bcad71ed2cc67c03c5e970e60ac6eb21db43679104431e678

Label Workers

Login to the master node again, and check all nodes are visible using kubectl get nodes. Notice that wokers have no role, meaning they won’t pick containers to run. Label them as workers:

kubectl label node k8s-worker1 node-role.kubernetes.io/worker=worker
kubectl label node k8s-worker2 node-role.kubernetes.io/worker=worker
kubectl label node k8s-worker3 node-role.kubernetes.io/worker=worker
kubectl label node k8s-worker4 node-role.kubernetes.io/worker=worker

Bootstrapping the Cluster: Part 2

Install MetalLB

MetalLB allows bare-metal clusters to have ingresses and load balancers without requiring a cloud-provider.

The first step to install it is to enable strictARP in kube-proxy. This can be done by running the command below:

kubectl edit configmap kube-proxy -n kube-system

Locate the line strictARP: and change its value from false to true.

Installing it is also simple:

kubectl apply -f https://raw.githubusercontent.com/metallb/metallb/v0.14.9/config/manifests/metallb-native.yaml

Finally, it needs to be configured in order to know which IPs it may use. In my case, the following is enough:

# metallb-ippool.yaml
apiVersion: metallb.io/v1beta1
kind: IPAddressPool
metadata:
  name: default
  namespace: metallb-system
spec:
  addresses:
    - 10.0.11.50-10.0.11.254
---
apiVersion: metallb.io/v1beta1
kind: L2Advertisement
metadata:
  name: default
  namespace: metallb-system

Apply the manifest using kubectl apply:

kubectl apply -f metallb-ippool.yaml

Install k8s_gateway

k8s_gateway exposes a DNS server that is able to resolve ingresses to nodes IPs. Installation is made through helm:

helm repo add k8s_gateway https://ori-edge.github.io/k8s_gateway/
helm install exdns --set domain=k.vito.sh --set service.loadBalancerIP=10.0.11.53 k8s_gateway/k8s-gateway

I want to expose 10.0.11.53 as the DNS service. My upstream DNS server forwards requests to k.vito.sh to this IP.

Install nginx-ingress

nginx-ingress is also installed through helm:

helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx

Apply the chart using the following command:

helm install ingress-nginx ingress-nginx/ingress-nginx -n ingress-nginx --create-namespace

Test the ingress and networking

The following manifest contains a deployment, service, and ingress that shows NGiNX’s default welcome message:

# test.yaml
apiVersion: v1
kind: Namespace
metadata:
  name: test-ingress
---
apiVersion: v1
kind: Pod
metadata:
  name: nginx
  namespace: test-ingress
  labels:
    app: nginx
spec:
  containers:
    - name: nginx
      image: nginx:latest
      ports:
        - containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
  name: nginx-service
  namespace: test-ingress
spec:
  selector:
    app: nginx
  ports:
    - protocol: TCP
      port: 80
      targetPort: 80
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: nginx-ingress
  namespace: test-ingress
spec:
  ingressClassName: "nginx"
  rules:
    - host: test.k.vito.sh
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: nginx-service
                port:
                  number: 80

Apply it using kubectl apply -f test.yaml. Eventually, the page will be available over test.k.vito.sh.

Enabling support for PersistentVolumeClaims

For this part, a new machine will be provisioned. I’ll use alpine as it is ultra lightweight, performs well, and installs even faster. I’ll configure it with 4GB RAM, 4 CPUs, and 100GB of disk.

Once the VM is provisioned, let’s configure NFS on it:

First, install required packages:

apk update && apk add nfs-utils vim

Create the data directory:

mkdir /data

Update its owner to nobody:nogroup:

chown nobody:nogroup /data

And configure /etc/exports just like in any other Linux server. It is important to notice that each node that will be given access to the NFS server must be included in the /etc/exports:

/data 10.0.11.1(rw,sync,no_subtree_check) 10.0.11.2(rw,sync,no_subtree_check) 10.0.11.3(rw,sync,no_subtree_check) 10.0.11.4(rw,sync,no_subtree_check)

After saving the file, reload settings:

exportfs -afv

And ensure to start the NFS service on boot:

rc-update add nfs
rc-service nfs start

Preparing Workers

Now, on each worker, install nfs-common through APT:

apt install -y nfs-common

And finally, install the provisioner through Helm:

helm repo add nfs-subdir-external-provisioner https://kubernetes-sigs.github.io/nfs-subdir-external-provisioner
helm install nfs-subdir-external-provisioner nfs-subdir-external-provisioner/nfs-subdir-external-provisioner \
  --create-namespace \
  --namespace nfs-system \
  --set nfs.server=nfs.kube.vito.sh \
  --set nfs.path=/data

Then, try to create a PVC:

# pvc-test.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: pvc-test
  namespace: default
spec:
  accessModes:
  - ReadWriteMany
  storageClassName: nfs-client
  resources:
    requests:
      storage: 20Gi

kubectl apply -f pvc-test.yaml

Reading the PVC status, everything should be good:

$ kubectl get pvc
NAME       STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   VOLUMEATTRIBUTESCLASS   AGE
pvc-test   Bound    pvc-ff3b4825-3836-4661-9958-c2b8948135af   20Gi       RWX            nfs-client     <unset>                 7s

TLS & Other Trinkets

What’s missing now is an Argo CD instance, and have automatic TLS for our services. This will be covered in the future!

Happy K8sing!