Any interest in reading in BX format?

[cvick32]

Hey, I just found out about the BX format that Embrilliance uses. I was wondering if there was any plans for pyembroidery to read in BX files and produce the resulting alphabet as a list of EmbPatterns. I would be interested on helping with this as I've been writing a tool that does a similar "typing out an embroidery font" thing as Embrilliance.

I know @tatarize has expertise in this area, so just wanted to see if there were plans to read BX.

Thanks

[tatarize]

Doing a little due diligence, from the sales copy and my knowledge of embroidery and various algorithms I've got to suppose that BX as a format doesn't have any embroidery in it. It's not a group of embroidery files or anything like that. It's simply the criteria and scale vector parts of the format. So rather than a letter A filled with stitches, you'd get what amounts to pathing that could be rendered into an embroidered A.

So rather than static glyphs with already present fills at different sizes, you get what is basically the skeleton of the A. It would need to load and save the direction of the satin stitches and the outline of the glyphs.

I've spent the last few days writing a parser for Hershey fonts specifically SHX files.
https://github.com/tatarize/shxparser

In that case what it's doing is what TTF and OTF do, they namely have a shapes stored in a lookup table the corresponds with graphics. So you type an O you get an O in the case of TTF this consists of at least two different paths of bezier curves that render into a raster O shape. You're taking lookup character information to make pixels on the screen and since the actual data in the font is a bunch of rasters you get pixels. But, if you make a much large letter, you maintain the shape quite well because vectors are scalable like that.

With SHX, these are line fonts and are getting important for things like lasers and CNC machines. Namely you have a single line. So rather than two different paths in an O where you have a small circle inside the larger one to make a sort of donut glyph. You would, with SHX and other Hershey font's just draw a line based circle. In the case of SHX, at least one of files in testing gave instructions for drawing a 360° circular arc (a circle, but with a start and end). The same thing must be true for correctly rendered embroidery fonts, but with some added complexity. What you need in a good font is nice satin stitches giving the directionality of the stitching, and the order in which those stitches go down.

Here's a quick visual for satin stitching in inkstitch.

The embroidery font, if done correctly, is not going to actually provide any embroidery, what you'll get instead is a series of glyphs that look like the above. You'd define the outlines, direction lines, center lines, and ordering. You'd then send that through rendering to make the stitches themselves at a particular size. So regardless what size of font you use, you'll get more stitching that gives you the correct density at those positions. This isn't a format that contains stitches it's a format that contains a few different datatypes that can be easily rendered into an embroidery with the right software/algorithm.

There's not really an easy way to go from a TTF where you get the filled outline to something that will necessarily work. You could with a bit of expertise add in a bit of information to an outline of a glyph to directly show the order the embroidery should go down as well as the direction of those stitches. There were some early embroidery TTF conversion stuff that tried to make fills for true type fonts that weren't very good. But, with a bit more info for a rendering engine you could explain how you want those fonts filled and save that as part of the font file.

I'd also need a couple examples of BX fonts which I couldn't manage to find (tatarize at gmail) in the time I spent looking, but mostly it's out of the range of this particular project since there's clearly no actual stitches within the BX format. Rather it'll be a different segments that are marked as outline, direction line, split line, and with a few properties for how it should embroider that font.

There's also some parts of embroidery fonts that would be even more engine specific. Namely the points for crossing over to the next letter. There are cases within TTF where there's special kerning adjustments for particular letter combinations. If, for example, OV are not made to be closer to each other than they actually are, they seem like they are much too far apart. So there's a table in the font to put particular letter combinations a bit closer or further apart. Likewise in Embroidery you'll have cases where you want to avoid cutting the thread, and you can have two different ways to render a letter, but sometimes in some situations you'll be able to cross the gap between letters more unnoticed than in some other cases. Ed for example would be much better to cross from E to d at the bottom but ET would be much better to cross to the T at the top, and admittedly many times you'll be rendering the text from the center out because of other various issue with pull compensation. But there is certainly a lot for a embroidery font engine to deal with (though in theory you could get pretty good results with a fairly naive system, get a glyph, get some direction lines, render the glyphs in order.

I could certainly take a look at BX as a format, but unless they are basically fake, they aren't going to have any stitches in them. They could rather have outline, different colors, direction lines, various lines to control the the satin stitch ordering etc. But, you're basically going to write a satin stitch engine, and store the satin data in a table that has 1:1 with various characters and a bunch of other interesting font tricks. In theory, if you were really good at fonts you could do things like map out Kanji characters which tend to make references to other base elements of the font, to different locations, at different sizes. And so long as your hints about directionality of the satin stitches hold up, it would be able able to render those characters.

[cvick32]

That's really interesting. Thanks for including some background information it's quite helpful. I just emailed over some BX files in different sizes.

Also, when trying to understand a file format do you look at the raw contents in a text editor, or is there some tooling around it? I've opened up some of the BX files in a text editor and I'm not sure how to begin parsing them.

[tatarize]

• The first section looks like it has a header, lists the available characters, has copyright info, etc.

• There's a sparse table that looks like it has the characters listed but doesn't seem to have much info, usually a number between 4-5 between them.

• Next there's something that looks like a bitmap.

• Next looks like actual data. But, running it through an analysis finds that it's pretty evenly distributed between the different characters, that implies that that section is either compressed or encrypted. I searched and it doesn't appear to have any zlib or gzip streams 

• There's apparently another table of Grp data.

---

There's a bunch of stuff to look at things with. A hex editor is massively helpful. Checking the number of times each character comes up can help find if things are encoded. There's also sometimes patterns that show up at different widths within data which makes checking those pretty helpful at times. Setting the hex editor to dynamic bytes per row and adjusting it can often reveal a things. And sometimes programs like Embrilliance will let you save out a BX file which is really helpful. You give it dummy information of basically a blank file except for one line. Then you do the same thing but you move the line very slightly. Doing that a bunch you can piece together what's going on.

You can also do things like download the free version of embrilliance that lets you use those fonts and just hack the file slightly and see if anything changed. Nudge it a bit that way and see what it's doing. Also, the NSA a few years back released Ghidra which is a pretty top notch bit of reverse engineering software. With know-how and educated guesses you can sometimes read the compiled source code well enough to figure out which parts do what, especially if it has a bunch of external hooks like you'd find in a DLL file. Sometimes when they have a EULA that basically says "use" that's even directly legal. Though the Embrilliance Express license does notably say "d. You will not decompile, reverse engineer, or create other software or products in any other medium based on the software." --- which means you need to find somebody else who would reverse engineer it, then publish how it works anonymously and you make whatever you're going to make directly from that description, without doing the reverse engineering yourself. Publishing speech gets some protections.

There's also a lot of insight with regards to knowing what the format can do, and what it's intended to do which can help a lot. Usually it's a lot of probing that can help considerably. Also experience, I spent months on end tearing apart various embroidery formats. There's a lot of redundancy where one bit of software is used elsewhere in a slightly different manner.

[tatarize]

Looking at the software and the interactions with the fonts. I believe that yes it is a font, but generally I don't think it's doing very much that is that special. It doesn't seem to be building the embroidery from scratch. The fact that it's quite clear that I'm not able to resize this stuff on the fly does actually suggest it's somewhat cheating. It very well might be just taking an embroidered set of glyphs and tweaking the geometry rather than building them from scratch correctly. The fact these fonts seem to have sizes attached to them and only work at particular sizes seems to suggest that we are dealing with static values.

The font nature of it is largely due to the later glyphs affecting future glyphs. You can select a particular glyph in a long phrase and rotate it and it rotates all glyphs after that. Also, rather than correctly adjust different letter combinations it seems to sometimes lack things like a lower case U and just does a big uppercase one for example queue looks pretty bad. I'm going to go ahead and say this isn't using an embroidery engine but is actually storing the static embroidery.

[tatarize]

I managed to get it to resize thing, it recompile or rebuild the font. It's legit just a bunch of glyphs with some sort of reasonable geometric manipulations.

The .be working file format is also going to be the same general format as whatever encoding the .bx files are in. So tweaking that stuff might not be too hard.

[cvick32]

Cool! Could you share the script you use to load in and do the transforms for the bx format?

Also, is .be an embroidery format? I don't see it mentioned anywhere in the README where it talks about the supported file formats.

[tatarize]

As part of their advertising and support for the format Embrilliance Express can be downloaded for free and you can load .bx fonts in them. Since it's the native software it seemed like it would be good for looking at what the restrictions and powers of the fonts actually are. So I did that and my new conclusion is that it's just fancy GUI work on top of a static character set.

The program in free mode also lets you save the files out. So you really could just take make a PES of every character or a giant one and split it apart.

The native format Embrilliance uses is apparently .be -- This isn't an embroidery format but a working format of an embroidery application. Usually those aren't worth converting (though PES is actually a working format with a PEC block which is embroidery format).

[tatarize]

Ink/Stitch has a pretty reasonable embroidery font format. And these formats can and do have auto-satin and static varieties of the same font. And I think there's even a script in there to convert from pre-digitized and auto-digitizing fonts.

The →.svg files contain specific layers which are inkstitch svg, and the json contains the typesetting information. So GlyphLayer-A contains the GlyphLayer for A. It's position is adjusted according to the kerning and pairs table and it's appended to the text. The typeset text is then put through the inkstitch rendering engine which converts the various marked satin stitches (and I'd assume fills, etc). And uses that data to produce the text at whatever scale is need. With apparently json flags for the largest and smallest scalings that should be done to the text.

So in that case we have a genuine example of prior art which also clearly permits both static pre-existing stitches as well as generated on the fly stitches. So it could be the case that the .bx files are relatively close to the same and the examples of the format does not include a font that regenerates from basic parts. -- Perhaps this is just a restriction on the .bx font itself, and not on the .bx font format as a whole, much as some of the fonts which could be contained within inkstitch are equally restricted, since they are originally static pre-digitized fonts rather than dynamic fonts that render in a post-processing engine thing.

I just checked and Embrilliance comes with a default font which permits resizing, called "Block", clearly when the font is resized it adjusts the positions of the particular nodes in a dynamic fashion.

---

You could load inkstitch fonts for your program with a couple key items that bridge some gaps. You'd need to load svg files with the relevant tags. Write that into embroidery and perform most of the typeset operations. I think meerk40t/svgelements could certainly load those things (though the view is set to none and that usually suppresses the loading that particular glyph, though I could hack it to overcome that (and probably need to)). And you'd need to read the relevant json parts for the font descriptions.

In theory if the fonts don't do anything really weird, you could get accessing at least the bad ones (static fonts with pure running stitches). And the good ones would be converted weirdly into embroidery of rungs/rails that were meant to be converted into satin stitches. I guess you could also reinvent the wheel a bit more and basically write your own inkstitch-satin stitch generator, as an intermediate step in the font.

Also, with enough research I guess cracking the .bx embroidery font format might be possible. But, would require knowing the engine behind Embrilliance font rendering. If this were done, in theory, you could try to convert between .bx and inkstitch-fonts. But, all of these would have somewhat similar criteria and might be a bit weird to house them all under the same roof. Embrilliance seems to have a leg up in the design department since they aren't just a hack on inkscape. They have genuine wysiwyg text editing, text on path, curves, slants, etc.

In block, there's also "stitch" options which allow you set a number of different fills. None, Length Limit, Tatami, Brick, Snake, Waves, Waves 2, Scales, Scale 2, Corn Row 1, Corn Row 2, Diamond 1, Diamond 2, Diamond 3, Chevron, Basket, Zig-Zag, Wicker, and Texture. These aren't really fills exactly, they are more like embossing on the satin stitch though.

The inkstitch algorithm generally for satin stitches is, I think, to walk the middle point between two rails in equal sized increments. So given two edges that are split into many pieces we walk along the edges until the center point reaches a threshold and we drop a zigzag in there. But, this means fonts using a similar algorithm will need fairly universally these sorts of classes of line segments. I think inkscape uses a metric that everything that crosses exactly two segments is a rung, and everything else is a rail, and in some cases where everything has 2 crosses like the lines in the symbol # for example, it decides the longest segments are the rails.

---

I think to code this up you'd either need something similar to inkstitch or if you'd want to switch software licensing to GPL3 you could just steal basically that code directly. Though it's not too hard to implement Bentley-Ottmann, and actually just directly store the segment types. So you'd actually have rail and rung as types. And write up your own engine for embroidery which is what rendering fonts would actually need. I only really mention this because I have some plans to do basically this in the somewhat distant future. Most of my current work is surrounding Meerk40t which is some laser control software, but I've written a lot of the parts of it with a future embroidery project in mind.

And with an actual embroidery rendering engine especially one which doesn't require a gui and is generally well defined and self contained you'd get situations where rendering a glyph at a particular size dynamically is both plausible and generally reasonable.

[cvick32]

Dang, that's really interesting. Thanks for sharing that, I'll definitely look more into how inkstitch is loading fonts and might try to get some of the ones I've already done in that format. I'll also try splitting the big pes file that comes out of Embrilliance to load fonts quickly into my program.

Overall, I think that dynamic sizing and typesetting of the fonts is currently outside the scope of both my expertise and needs for my project. I'm mainly helping small embroidery businesses speed up their workflow (they're used to clicking and dragging each letter file into a program like PE Design and then typesetting). So, as long as I can format the fonts they already use in a reasonable amount of time, I'll be good.

Thanks again for your help.

[tatarize]

@cvick32 Also, check this out:
inkstitch/inkstitch#1312

I think it's entirely possible to use the script or at least get close to quickly converting the static glyphs into inkstitch. At least then it would mostly subplant the core reason there and let you make a quick font out of the font files you have and quickly turn that into embroidery.

[cvick32]

https://inkstitch.org/tutorials/font-creation/

Leaving this here as a note for how this is done manually in inkstitch.