hckrnws

If an integer is drawn from an n bit space, it has n bits of entropy. Yes it really is that simple.

That's not true unless the distribution is uniform. If it were always true, then compression would not work. Compression, in a simple case, often compresses data stored as byte sized chunks into smaller space, precisely because those 8 bit integers do not have 8 bits of entropy each.

Yes; plus the caveat that this draw has to be "truly" random, and have a uniform distribution.

How does this relate to the mean entropy of 1 noted in the OP article?

Having ripcorded out after realizing the author was trying to prove that water was wet, I'll assume that it's "normalized entropy", in a range of 0-1, indicative of the distribution across the space.

[dead]

The comments section on the author's book 'How to not be wrong' is one of the best things I have read in ages. I am so glad the author left it public. Imagine releasing a book called 'How to not be wrong' and you have like 200 people telling you that you are wrong. Posting in the comments section of your minimal personal blog.

This makes some jumps I was unable to follow.

like, the part where they get a_i log p_i , well, the sum of this over i is gives the number, but it seemed like they were treating this as… a_i being a random variable associated to p_i , or something? I wasn’t really clear on what they were doing with that.

I asked OpenAI.

Take an $n$, chosen from $[N,2N]$. Take it's prime factorization $n = \prod_{j=1}^{k} q_j^{a_j}$. Take the logarithm $\log(n) = \sum_{j=1}^{k} a_j \log(q_j)$.

Divide by $\log(n)$ to get the sum equal to $1$ and then define a weight term $w _ j = a_j \log(q_j)/\log(n)$.

Think of $w_j$ as "probabilities". We can define an entropy of sorts as $H_{factor}(n) = - \sum_j w_j \log(w_j)$.

The mean entropy is, apparently:

$$ E_{n \in [N,2N]}[ H_{factor}(n) ] = E_{n\in[N,2N] [ - \sum_j w_j(n) \log(w_j(n)) ] $$

Heuristics (such as Poisson-Dirichlet) suggest this converges to 1 as $N \to \infty$.

OpenAI tells me that the reason this might be interesting is that it's giving information on whether a typical integer is built from one, or a few, dominant prime(s) or many smaller ones. A mean entropy of 1 is saying (apparently) that there is a dominant prime factor but not an overwhelming one. (I guess) a mean to 0 means dominant prime, mean to infinity means many small factors (?) and oscillations mean no stable structure.

one of my favorites

https://xkcd.com/221/

Around 2001 I was working at Broadcom's networking division in San Jose. The switch chip we were working on (10Gbps x 8 ports) was understaffed and we hired a contractor at $120/hr to do verification of the design. He was pretty young, but he came across as confident and capable, and every weekly meeting he was reporting good progress.

Unfortunately we weren't reviewing his work, just trusting his reports, as we were overworked getting our own parts done. After a three months of this I said to the project lead: something smells wrong, because he hasn't filed a single bug against my design yet.

So we looked at his code, lots of files and lots of code written, all of it plumbing and test case generation, but he hadn't built the model of the chip's behavior. At the heart of it was a function which was something like:

    bool verify_pins(...) {
       return true;
    }

My boss and I literally slept at work for a month, with my boss building the model and fixing other TB bugs, and I addressed the RTL bugs in the DUT as he found them.

[dead]

In John C. Baez "What is Entropy?" (A 122 page PDF best suited for reading on airplanes without wifi and in flight entertainment), he states:

> It’s easy to wax poetic about entropy, but what is it? I claim it’s the amount of information we don’t know about a situation, which in principle we could learn.

The entropy of a random integer being 1 makes intrinsic sense to me, given I didn't spend years in theoretical math classes