Category Archives: Technical

Python is addictive

Been a while since I’ve felt this way, but Python brought it back out. Python programming is pretty addictive. I find myself pondering the next little Python ditty to whip up, and going through  withdrawal without it. Gotta get my Python fix!

If you’re a developer who can taste a good design, a good implementation, you know what I mean.

Of course, there are plenty of things I wouldn’t write in Python.

 

Downsides of Collaboration

Here’s an outstanding video on how collaboration can not only kill creativity, but dupe our very perceptions. Steve Wozniak:

Most inventors and engineers I have met are like me: they’re shy and they live in their heads. They’re almost like artists. In fact, the very best of them are artists. And artists work best alone where they can control an invention’s design without a lot of other people designing it for marketing or some other committee. I don’t believe anything revolutionary has ever been invented by committee. If you’re that rare engineer who is an inventor and also an artist, I’m going to give you some advice that might be hard to take. That advice is: work alone. You’re going to be best able to design revolutionary products and features if you’re working on your own. Not on a committee. Not on a team.

And it gets better from here.

You’ll undoubtedly apply it to situations closest to your heart.  It resonates with me and the software I write. Of course we can’t just make up our own requirements, but the final product needs to come from you.

I also hear a call to courage. Don’t be arrogant, but stand your ground. Use your best judgment. Don’t be dulled–or let your project be dulled–by the strongest personalities in the room.

Thoughts?

 

 

Fantasecond response time

Here’s a fascinating in-depth study of one second of market data for a single stock:

HFT [High Frequency Trading] Breaks Speed-of-Light Barrier, Sets Trading Speed World Record.
Adds a new unit of time measurement to the lexicon: fantasecond.

On September 15, 2011, beginning at 12:48:54.600, there was a time warp in the trading of Yahoo! (YHOO) stock. HFT has reached speeds faster than the speed-of-light, allowing time travel into the future. Up to 190 milliseconds into the future, or 0.19 fantaseconds is the record so far. It all happened in just over one second of trading, the evidence buried under an avalanche of about 19,000 quotes and 3,000 individual trade executions. The facts of the matter are indisputable. Based on official UQDF/UTDF exchange timestamps, there is unmistakable proof that YHOO trades were executed on quotes that didn’t exist until 190 milliseconds later!

Millions of traders depend on the accuracy of exchange timestamps — especially after bad timestamps were found to be a key factor in the disastrous market crash known as the flash crash of May 2010. …

The lowly time-stamp. Not rocket science. Not crucial like price, right? What’s to get wrong? An exchange with unlimited resources couldn’t mess up like that, could it? (Assuming the analysis is correct.)

As a software builder, I wonder how did it happen?

It probably wasn’t buggy code: unit tests are good, but more of them probably won’t catch this one.

Was it sheer load? It wasn’t the highest overall traffic they experienced:

The chart below shows quote message rates for UQDF and multicast line #6 which is the line that carries YHOO quotes. … Although traffic from YHOO was a significant percentage of all traffic on UQDF, it was not high enough to indicate any problems. Note the much higher surge on the right side of the chart; there weren’t any known problems at that time in YHOO.

So the system would pass a generic stress test. Yet still allow this.

Maybe the system was stressed in some unexpected way. This stress’ profile wasn’t predicted, characterized, mitigated. The system design didn’t account for it.

A multi-tasking operating system is likely part of the picture. Maybe another task was started that tied up crucial resources, even just CPU. Maybe not even a lot of resources, or not for very long. A multi-tasking operating system is geared to never say “enough.” Another task in the CPU’s ready queue? No problem. Out of RAM? Swap some to disk: user programs can’t even tell!

Information hiding gets us off the “honors system” with memory usage. But in the time domain, you’re still very much on the honors system: no one can make you complete your task in a given time. In fact, any innocuous-looking call can have any amount of code hiding deep inside it.

You can make a user wait for a tenth of a second or more and still seem really responsive. (Think “garbage collection.”) But a real-time data feed won’t wait. It’s a different ball-game.

Maybe we’re just seeing a hardware limitation hit. Maybe the best machines money can buy just can’t keep up with certain kinds of volume. There’s no guarantee they can.

Interesting stuff. Hat’s off to the author for such a detailed analysis.

Via ZeroHedge.

LinkedIn broken?

As of right now, a couple LinkedIn connection requests I made, that were accepted, still aren’t showing up in my connections list. I received the e-mail that we’re connected. So it’s half-working, half-broken, even after 30 hours or so. (Surely the electrons can travel from one end of LinkedIn to the other in that time.)

The mind boggles at the mess this could potentially be if they’re dealing with a breakage of something that fundamental. My heart goes out to them. I hope they have (or can create) the right view of their system to make it all good. I haven’t seen any announcement to that effect.

Avoiding SubInACL.msi

Problem: An app I wrote (some time ago) refuses to run on Windows 7. MFC’s CDialog::DoModal() returns immediately. The web suggests it’s using an unregistered control. Sure enough, the registration for a control fails with:

The module “msflxgrd.ocx” was loaded but the call to DllRegisterServer failed with error code 0x8002801c.

One person says get and run SubInACL.msi, and use it to make sweeping recursive permissions changes to the registry and system directories.

Yikes! Really?!?!?

This much-less-intrusive solution worked for me:

Right-click on C:WINDOWSSystem32cmd.exe, and choose “Run as Adminisstrator”. From that command-line, run the registration:

regsvr32 msflxgrd.ocx

The registration now passes, and my program now comes up (not needing to be run as administrator).

Note that we thought the account we were running under had full Administrator privileges, but still failed to run this app as such. (It didn’t ask for the password when running cmd as administrator.)

Bye-bye manifests!

Here’s a very interesting development, brought to my attention by a fellow developer:

Microsoft is doing away with the manifest scheme in Visual Studio 2010. From Deployment in Visual C++ 2010:

Differences between Visual C++ 2008 and Visual C++ 2010

The most significant changes between Visual C++ 2008 and Visual C++ 2010 are:

* Visual C++ libraries no longer depend on manifests and are no longer installed in the WinSxS folder.
* Dependent applications and libraries no longer require manifest information.
* Local deployment no longer requires a satellite manifest.

To me, the manifest scheme just traded one problem for another. And the problem traded away (DLL issues) was one I never had.

Too Simple to Fail?

I’m doing some low-level Ethernet work (TCP/IP), with a device connected through two plain vanilla network switches. (I’m testing breaking the connection with neither side detecting the link-lost condition.)

I made a change to my software, and suddenly the Ethernet is dead. It seems to send and receive nothing.

I back out my change, and it still doesn’t work.

I back out a few more: no effect.

As it turns out, one of the Ethernet switches got confused and quit routing correctly (even through this simple configuration).

You can tell that the switch has some smarts:

  • When you look at the traffic with WireShark, you don’t see what’s exchanged between the switch’s other nodes.
  • When you have a mix of GB devices and 10MB devices on the same switch, the gigabit devices transfer data at full speed, so you know the switch isn’t exposing every packet to every (10MB) connection.

Still, I was astonished that this could happen.

For the record, the switches were:

  • Netgear ProSafe 5 Port Gigabit Switch (Model GS105)
  • Netgear 4-Port 10/100 Dual Speed Hub (Model DS104)

The older DS104 was the prime suspect. I simplified the route to just one switch (the GS105), which got it working. So I cycled power on the DS104, reconnected the original configuration, and everything worked again.

Update 4/8: They say you never learn anything from a mule’s second kick. I unceremoniously decommissioned the DS104 after it did the same thing again, this time with no provocation.

UDP Broadcasts Wrong Source Address

Symptoms: Windows machine with two network interfaces (NIC), 10.x.x.x and 192.168.x.x. I send a UDP broadcast message (to destination address 255.255.255.255), and see the same message go out on both networks. The problem is, the source address that goes out with each packet is the IP address for my first network interface, so it’s sending an unreachable address out my second network. Responding devices start doing ARP requests to try (in vain) to find that unreachable address instead of responding. (Watch it with WireShark.)

Cause: 255.255.255.255 is too general. TangentSoft has the general solution (implemented in boost). I craft the broadcast address more carefully, and it works.

	10.x.x.x (subnet 255.0.0.0) => 10.255.255.255
	192.168.1.x (subnet 255.255.255.0) => 192.168.1.255

It’s not the language or library. It’s seen for C#, C, and C++, etc, on Windows Vista, and here.

The broadcast address 255.255.255.255 is also known as INADDR_BROADCAST or ‘‘ in Python.

You only need one socket, which you can bind to all interfaces.

Enumerating your network interfaces requires some work: find it under Win32_NetworkAdapterConfiguration, fields IPAddress and IPSubnet. Those fields are arrays: you want element zero.

If you’re using python, you need the WMI package, which is built on the PyWin32 package.

import wmi
nicsList = wmi.WMI().Win32_NetworkAdapterConfiguration() # your network adapters

If you’re getting WinSock error 10013, you need to set SO_BROADCAST.

Hope that starts you down the right path.

Also, if your devices aren’t configured for the same subnet, crafting a sub-net-specific broadcast message means you’re not going to find them. For example, if you have a 10.x.x.x device off your 192.168.x.x interface, it’s going to ignore your 192.168.255.255 broadcast. You’ll have to fall back to disabling a network interface and using 255.255.255.255 to get to it.