Hardware: My laptop (Thinkpad T14 G1 16 GB): It’s what I have. Even if hardware prices weren’t astronomical I would use it for figuring things out. Here’s to hoping that by the time I know what I’m doing the price for edge computing dev boards will come down so I can build a little box with the knobs and the blinken lights and so on.
Language: C++: A few years ago, I would have definitely picked Python for rapid prototyping. I’m very familiar with Python and can be very productive in it. I’m not sure why I’m leaning towards C++. I guess I’m worried that I will come to some critical juncture where the Python code is too creaky and too large and a bunch of things have to be coded as C++ libraries any way. I might rethink this …
Framework: llama.cpp: It’s early days. I just want to get up and running with an audio or visual loop. The next stage is to see if the framework is flexible enough to allow me to reach in and sample internal computations and tweak weights. (I initially looked at LiteRT-LM, but the docs weren’t great)
Incidentally, llama.cpp was what Google Search Gemini suggested I do too
when I described my use case. It gave the following arguments:
ggml under the hood.
Because it is written in low-level C, you can directly print, log, or export
the buffer pointers containing the activation values during the forward pass.llama.cpp or the graph
evaluation code in ggml.c).Ideally, I would rather not modify the library code itself. I would prefer either hooks (callbacks, say) or an API that allows me to examine the activation values after each step through of the graph.
As notes for the next step
examples/debug for how to add callbacks/hooksexamples/deval-callback for callbacks