I've been meaning to write something about file format design for a while, and might as well do a blog post. This is pretty unorganized, in a moment thing, so take it as it is.
There's a few questions you have to ask. Different files have different goals, after all.
It's a good idea to check what the prior art is, as it's more than likely that there's already something out there that fits the bill. Let's say you need an image format. There's tons of image formats out there, so one of those might already solve your problem, possibly with tool integration and plenty of library code to boot.
But it's also possible you're in need of a image format for something really specific, like a 4-bit cpu driving a custom LCD with a strange pixel order pattern. Hitting that with jpeg is probably not the way to go. So you need something custom. Assuming, again, that the community for that specific use case doesn't already have a standard format. That you don't have a problem with.
There are cases where you may want to use existing complex systems, such as SQLite, or something that automatically serializes your data; some programming languages even provide functionality for that. There are pros and cons for these approaches, so you may end up having to design something from scratch.
Parsing text is a pain, so again you need to ask whether you can get away with json, xml, ini or some other commonly used format, and just design fields in it. Just picking one of these formats isn't the end, though, since the structure of the data matters.
There may be a reason you don't want to use those formats, in which case you need to make decisions based on your specific use case, and the reasons why you can't use a common format. In this case I would concentrate on minimizing potential errors, both technical and human; Make the format as easy and foolproof to parse as possible to avoid plenty of headaches ahead (and you may want to avoid fscanf for its quirks), and make the human-editable data as unambiguous as possible: if you need true/false values, pick one set and stick to them. I would use 0 and 1 as it's harder to typo "0" than "Disabled".
If the data doesn't need to be human readable, it's often way easier to make a binary format. A common structure for these is a "chunked" format used by various file formats including 3d studio .3ds files, autodesk animator .fli/.flc, EA's .iff (famous on amiga), Apple's .aiff, Microsoft's riff (including .avi) and countless others. The basic idea is to define data in chunks, where each chunk starts with two standard fields: tag and chunk length. Chunks may also contain other chunks, meaning your whole file can be one chunk that then contains other chunks inside it. You can find the whole file size by just checking the root chunk size.
Some people call the initial bytes "magic numbers", used to identify the format. I just call it the tag of the root chunk. Having the whole file be a chunk has the additional benefit of being able to concatenate your files into one bigger file, if need be.
As an example, let's say you have a file format about cats. You might start your file with a "cats" chunk, meaning the first 4 bytes in your file are "cats", followed by 4 bytes describing the whole file length. After that you might have some header fields such as format version, how many cats are described in this file, maybe a field for identifying what software generated the file, etc. After these, you'd have a bunch of other chunks like "toys" for different cat toys (again starting with the tag, followed by that chunk size), "name" for ideas for cat names, "colr" for different cat color schemes, and so on.
Using a chunked format allows chunks to be in different order, and there may be optional chunks, possibly added in later versions, adding backwards and forwards compatibility. Which leads us to:
Whether your data is in text or binary format, design it so that it's possible to write different tools that read your format, that may be interested in different parts of the file. If I wanted a corpus of 8-bit palettes used in .fli animations, I could just write a tool that skips all chunks until it finds the palette chunks and extract just those.
It doesn't matter whether you never, ever, ever plan to change the format, having a version field in your header doesn't cost much but can save you endless headache down the road. The field can be just a zero integer that your parser ignores for now.
Even if it's for personal use, write a specification of your format, in a way that someone else than yourself could reimplement tools for the format. That someone else may be yourself in 20 years, and I guarantee you, you won't remember all the details. Be as unambiguous as possible.
Most formats out there have these. Save them for a future version of the format. You may be tempted to pad out the file header with reserved field for future expansion, and that may be valid. However, it's cleaner to have a field in your header that states where the first sub-chunk starts; that way you can expand your header as much as you like in future versions, with old code being able to ignore those fields and jump to the good stuff.
If you're targeting a small device (something embedded or maybe retro), the target hardware may set a lot of restrictions on your file format design. Anything can compute anything, given enough time, but if your device is based on big-endian 16 bit integers, it's probably good idea to stick to that instead of making all fields 64 bit little-endian. Other limitations may also apply, such as memory sizes and alignment.
Depending on your use case, compression may be a factor. Should you compress your whole file, or do a chunk at a time? Are you planning on modifying the data later on, or is a created file always considered read-only? How much memory does it take to store the whole decompressed data? Do you need to seek within the file, or is it always read from beginning to end? How fast does the decompression need to be?
If speed is a factor, it's possible that compression actually speeds up reading, as there are compression algorithms out there where the time saved reading less data from storage is greater than any slowdown caused by decompression.
You may want to look up whether the filename extension you're deciding on is in use already. Most extensions have three characters, which means the search space is pretty crowded. You may want to consider using four letters.
It's been a while since I last wrote anything for the zx spectrum, or done any creative programming in general. I've concentrated more on my steam backlog and reading books and stuff. I had played through Dredge (as much as it's sane to do so), saw that Ludum Dare had "depths" as a theme, and, well, the Comp.Sys.Sinclair Crap Games Competition rolled around, so I figured.. I might do a fishing game.
I started off with a design document. Here it is, verbatim:
space to fish - line goes down, release, line goes up: get depth value (0..32)
16 or 32 different fishes:
- 4x4 random pixel pattern
- 1-8 rando length
- random depth offset
- valid depths random(1,2,3,4, 6,8,14,20)
Each depth has chance to get fish that is PoT available fish + nothing
(if 2 fish available, always get a fish; if 5 fish available, 5/8 chance to get a fish)
Clients come with shopping lists of 1-8 fishes (with pictures)
Player discovers fishes and has data of what fish they have caught at what depths
Game goes on forever
I did not follow the plan quite one to one, but the gist is there. Next up, I did a mockup in my old Photoshop CS5, turned that mockup into assets for the game, and then just had to find motivation to actually code the game.
I used my Image Spectrumizer to convert most of the assets from Photoshop to the speccy. The moving bits (checkmark and faces) I converted with a simpler tool that can be found in my speccy github repo. One awkward consqeuence was that the stuff produced by the spectrumizer had ink and paper reversed compared to the other stuff (including the ROM font). I could have spent time fixing that, either by making a tiny tool to invert everything, or maybe do that at runtime, but opted to just xor 255 the stuff when drawing to the screen. It's fast enough.
The line going down and up is done with putpixel. Doing a putpixel on speccy's weird screen is rather convoluted; first you have to calculate the address if the byte the pixel exists in (the address is a bit swizzled: 0 1 0 Y7 Y6 Y2 Y1 Y0 Y5 Y4 Y3 X4 X3 X2 X1 X0), and then do a read-modify-write on the byte. Oh and there's no barrel shifter on the z80. I opted to do a 8 byte lookup table. In a stroke of inspiration I just made the putpixel use xor to put the pixel, so the same call can be used to undo the effect.
I didn't feel like reinventing a random function so I found one on the net somewhere. I also struggled with the coordinate calculation for some reason, so I found a ready function for that as well. There's two routines for it in the source code now - one that calculates 8x8 coordinates and one that calculates 1x1 - the 8x8 one is my original code I wrote for the z80 tutorial.
Code writing wise, I first did everything that did something visual; every element has its own function. The tick is a 8x8 pixel element, the bobbing rod is a 8x16 one, the faces are 24x24, and text output is another one. And since we need numeric displays, that's its own routime (although it uses the text output routine for actual output).
The input is literally a "any key" routine. It checks if any key on the speccy is pressed or not. I also considered adding Kempston support, but if you don't have a Kempston interface, the port may have random values in it, so I left it out.
For audio, I didn't even consider using some third party bleep routine, but rolled my own. The audio requirements are fairly minimal, after all.
While doing the game logic I figured that the game is annoying enough without having a random chance of not getting a fish, so that was out. There's depths where you can't find any fish, anyway. Having fish that only exist in one depth was also super annoying, so that was out. As a final adjustment I added the "top" and "deep" hints for the fishes that have depth < 10 and depth > 20.
I used my mackarel to turn the raw binary output from the assembler into a tap file along with the loading screen and stuff.
I tried to find testers for the game, but only found a couple of people willing to give it a spin, and the feedback was basically "it works", so that was that. I found some bugs in my own testing and fixed those, but never received any other bug reports. The excelent writeup in csscgc suggests there are bugs, though. Oh well.
You can find the entire z80 assembly source code here on github, and if you want to take the game for a spin, the game .tap file and online emulator are here on this site.
So I decided to play through the Mass Effect trilogy again after playing Andromeda. And I was downright floored just how much better those games are. Each of the games has way, way more variety than Andromeda does, even though Andromeda allegedly has more voice lines than the whole trilogy has.
There's little I have to add to what I said earlier. Still love the trilogy, still enjoyed Andromeda, still feel the couple novels I've read were good. I also ordered the comics, which are probably the last of Mass Effect I'll see. There may be a TV show being produced, but it's by Amazon - likelihood that I'll watch it is slim. Fifth Mass Effect game is in preproduction at BioWare / Electronic Arts, with approximately zero people from the original trilogy team, so I don't have high hopes for it.
Assuming they manage to make the new game, I fear it will be in the same risk-averse stuff as Andromeda, which I feel is not a Mass Effect game, but something else with a Mass Effect coat of paint. Kinda like Beverly Hills Cop 3..
Anyway, feeling a bit empty now. All those goodbyes. Sigh.
In other news, the new year resolution progress:
I have strong suspicion that caffeine causes migraine for me. Let me explain.
Some years ago I ended up in a situation where we had frequent (and consistent) visitors for whom we'd make coffee. Since we don't drink coffee ourselves, pre-ground coffee would go bad, so the most sensible thing was to buy beans instead. And once you have beans, you need a proper coffee grinder, and might as well get an Aeropress. And after all this investment, you kinda have to learn how to make proper coffee.
So I did. We picked one light roast bean type that was readily available locally, and I tuned the grind size so that it wasn't too acidic or too bitter for my taste. Wife was also surprised to find that there was coffee that she liked.
That lead to a routine where I'd drank one cup of coffee a day. Not excessive by any means. That went on for maybe a month, and then the migraine started. I'd stop drinking coffee, migraine went away, tried a cup, and bang. So it was somehow related to the coffee. I measured my blood pressure, got bloodwork done and my medication was changed, but it was still pretty weird that coffee would cause this. Coffee does rise your blood pressure for a brief time, but the amount I took shouldn't make a difference. Liquorice also does, and that has never caused problems for me (that I know of).
Fast forward a couple of years, and instead of a candy advent calendar I figured I'd try a tea advent calendar instead. Which means a cup of tea a day. See where I'm going with this?
On day 21, migraine. Day 22, migraine. I stopped drinking tea, no migraine. Haven't drunk tea since, haven't had an attack since.
This makes me wonder. I've had a cup of coffee or tea occasionally without any problems, but these two "cup a day" experiments have lead to migraine attacks. Looking back to when I've randomly gotten an attack, has caffeine been part of the equation? Perhaps? When I was a kid and got very severe attacks pretty often, I was drinking a lot (and I mean a lot) of coca-cola. The attacks ended when I was doing my military service, so I've attributed it to getting in shape physically, but.. I also had a lot less caffeine during that time.
Who knows? The best I can do is just avoid routinely drinking caffeinated things.
Mass Effect: Andromeda came out in 2017, so I think I'm allowed to drop a few spoilers here, not that they really matter in your potential future enjoyment of the title.
I loved the original Mass Effect. For the first time in a very long time, the game gave me what Tim Schafer calls "promise of infinite possibilities". At the beginning you couldn't "feel" the edges of your playground. You did find them pretty soon, but still, that was quite the feat. With the "living the movie"-like feel, great writing, great music and, for the time, great graphics, it just grasped me. The fact that it was basically modern-day Star Control 2 (the #1 game of my formative years) may have played a part.
Then they said that they "compiled a list of player's feedback and addressed every single one" while making Mass Effect 2. I felt that was a mistake.
Well, ME2 was much more "game" than an immersive experience. Sure, many of original ME1 mechanics were, in retrospect, rather clunky, which was very apparent to me when I re-played the game later on.
ME3 was only available on EA's store so I didn't play it for a longest time. But I did buy a couple of ME novels by the original game writers. They're a fun read.
Anyway, ME3 was released with some complaints, especially about the ending. Apparently someone in the management felt that the original plans for the plot, which made sense and were good, were "leaked" or something, so they had to pull something new out of the hat and the ending was weird. They patched the game later on to make the ending more sensible (I think?), and the version of ME3 that I eventually played was the legendary edition, so I never saw the original ending. ME3 is not a bad game, but it's lesser than the original, or even ME2.
Then Mass Effect: Andromeda was released, to very negative press. I largely ignored it. It was on sale for a fiver over the holidays, so I figured, what the heck.
I've been trying to form some kind of idea of what went wrong with the game, and figured I'd write a blog post about it.
First off:
It's.. content.
The first impression of the game comes from the graphics, or more precisely, faces of the characters. They're.. ugly. It feels like it's impossible to make a pretty character in the character editor. I don't know what went wrong there. You kinda get used to it. As space visuals go - I mean planets and such - it's very pretty.
The writing feels like a fan fiction of Mass Effect. There are some pretty good or pretty funny bits here and there, but you can't have a game of this size without hitting the mark sometimes. Since you have to have some shocking things, there's a religious character in the game. Not in "scifi religion" but real-world religion. Like most things in this game, it never goes anywhere. Elsewhere, alien lore is just dropped on your lap out of nowhere, used in one side mission and then never referenced again. Some writer apparently fell in love with "colors we don't have names for", since that phrase stood out several times.
There's several story beats that apparently had to be in the script because they were so nice in earlier games. Except they're very watered down. Case in point:
(and here's the more majorly spolery bit)
In ME1 you have a couple of squad mates who you have been building relationships with over time, maybe even romantical ones, and at one point you have to choose which one lives and which one dies. It's quite dramatic, and comes as a shock. The one who dies will no longer be there, and your choice has literal consequences. The surviving character even pops up in ME2 and ME3.
In ME:A you're doing a mission, meet someone during that mission, and then are asked to either save that character's life or someone else you haven't met. I haven't played the alternative choice, but there's a couple of lines of dialogue from someone complaining about your choice and that's about it.
There's also a "you must disobey the authority to save the day" story beat in ME:A as in earlier games, which pretty much comes out of nowhere, and results in no consequences whatsoever.
(end of spoilery bit)
Overall, I don't know if any of my choices changed anything in ME:A, even though the game claims that they do.
At around 15 hour mark I realized I don't even remember a single character's name nor do I care about any of them. I mostly remembered doing 3d platforming and sudoku puzzles.
The game has "only" two new alien races even though you're in a new galaxy. I'm fine with that, and there's even some story justifications for it. (The same justification can explain the identical fauna on all the planets). What does bother me is that the aliens are not on par with the old ME alien designs.
Technically it's pretty obvious that different worlds were implemented by separate teams. Some things that should be common just work slightly differently in different places. Why are doors so slow on one world? Why is there a text menu to pick where to go, while everywhere else you'd just use a different door or something? This is definitely nitpicking, but these would have been super easy to fix.
Another problem with the game is that there's just so much stuff in it. You assign strike teams to do something that takes real-world time to do. I don't know why, or what the benefit was. There's mining. There's cryo pod assignment stuff. There's crafting. There's research. There's collect-a-thon of a billion different things that don't matter. There's planetary investigations from orbit (click on planet, wait for transition, click on planet, maybe find something, click on planet, move to the next one). The game would have been better off by trimming all that unnecessary crap. Maybe leave the (very pretty) planet orbits, flesh them out a bit.
Gameplay wise it felt more like Assassins Creed than Mass Effect. Pick point of interest on the map, go there, pick something up, press a button or shoot a guy there, move to the next one. There's no towers to climb (well.. sort of) but there might as well be. This sounds like a complaint, and it kinda is, but then again, I enjoy this kind of gameplay. It's just not what I want from an ME game.
You're driving around a lot. A lot. I actually used a cheat to get infinite boosts just to make that more tolerable. Didn't use the cheat for anything else. I also played at easiest difficulty level and ran through many of the fights the game was setting up. Turns out, most of them are optional. (Towards the end of the story missions there are some places where you run past enemies just to find a door that doesn't open before you go back and shoot them, which makes zero story sense). Even with these shortcuts the game has still taken me some 50 hours to play, and I think I'll still go back to finish a few side missions.
Yes, the game doesn't end at the credits. It's one of those sandboxes. Which means the story can't change a many aspects of the world. Nor can the side missions. They're just content that can be played in whatever order.
As I'm writing this I have ME soundtrack playing, and realized that in ME:A the soundtrack is.. just there at the background somewhere, and doesn't make a fuss. That's pretty much what the whole of ME:A is; it's ME themed, but doesn't want to draw attention to itself.
It's a new year; let's hope it's more positive one than the dark clouds in the horizon seem to predict.
And here's the new year demo:
And here's a reminder that my Ko-Fi shop exists
I don't have the greatest track record when it comes to new year resolutions, but here's my sketch for next year:
Apart from all that, I have this annoying idea buzzing in my head that I should pick some faraway city, and then obsess about its history like only someone who has never been there can, and then spin a game or book or something on a fantasy setting on top of that. Maybe use all of these story snippets I've played with over the years. Who knows.
Likelihood that I manage all of the above? About zero. But maybe something will come out of it.