June 22, 2022

A Data Problem

"We have a data problem," Ines said.

"That's why we have an Ines," Kapri said.

"I know. But this doesn't feel like repairing an archive or routing traffic. This feels like a dam that's starting to crack."

Kapri put her tablet down. It wasn't like Ines to be poetic about data. "What are we looking at?"

"We brought the media archives with us, the ones with all the old Earth media we could get. That was all pre-indexed, sorted, and it still takes up a rather large brick of crystal storage. But what's nice about it is that everything we go looking for is already in there. If I send you a media link to some song or something, I'm just handing you a pointer to a spot in the crystal to read that data from."

"So is there a problem with the media archives?"

Ines shook his head. "Not exactly. They are fine. We just got a new one."

"A new one?"

"A big one."

It clicked. "The Orb. Right. Of course, it's a giant pile of data. Right?"

"Yes. But it's not.. human data. I don't mean that in a pejorative way, I mean, it's not stored and organized the way we would make sense of it. And it's so much denser, in terms of information, than any of our crystals. It all fits in that Orb, somehow. And that's fine by itself.

"Except now pretty much every single person throughout the entire fleet is spending most of their time searching through it. And clipping bits of it to local storage. And creating new media based on what's in there." At each step, Ines hands grew farther apart, emphasizing the growing scale of the problem. "We just connected to more data than we know how to store, and we're going to run out of digital memory just unpacking it."

"Why don't we just use index links, like we do for our media?" Kapri asked.

"Two reasons. The first is, we don't really have an index. The Orb acts as a search daemon, it translates our request and navigates its own internal data structure, then it sends human-readable results to us through the interface circuits. For an API, it's effective, but not very robust. The best we can do right now is codify search requests and append a little direction to help narrow down the target data. But that's like saving bookmarks of your Google searches, it's messy."

"Ok, not great. But it still lets us explore without unpacking everything," Kapri said.

Ives shook his head again. "That's the other reason. The data comes in this fractal format. Concepts are built out of complex webs of relational data, every time an idea is referenced, the full data about that idea is included as part of the reference. So even a simple search returns a complicated nest of ideas to unpack and organize. The Orb will connect several levels of information for each reference before arriving at a human word it trusts us to understand well enough. Even if we're looking for just a small piece of data, we're importing a huge chunk of archive just to know what we're looking at."

Kapri sighed as the problem grew into full view. "We can't exactly tell a fleet of explorers not to keep looking. We're going to need more space. Can the Orb help us with that?"

"Maybe, but not directly. One of the topics that seems to be off-limits right now is anything about the internals of the Orb. We are of course trying to learn what we can from the tech described in the archive, but a lot of the focus is just on building an organization framework. Marshall showed me a diagram for writing data to the oscillations of a small gravity well. He thinks maybe each ship's gravitational wake could act as storage. That's just one of like, 6 groundbreaking things I was shown this morning. But we can't just build one, we have to understand their understanding of gravity, we have to learn how to parse the circuitry, we have to learn how to even synthesize the materials to make the circuits. It's going to be a while before we figure out a good order for integrating a lot of this."

"I see what you mean about a cracking dam. Do you have any ideas on how else we can keep this from spiraling out of control?"

Ives looked at the ground. "Yeah. But it sucks. We have to start data rations."

Kapri winced. "Ugh. Yeah, that won't be popular. Have you built any models? How harsh are the limits we're looking at?"

Ives swiped through his tablet to bring up a few charts. "For starters, we need to preserve space for future, non-Orb data. We can't fill up at the start of our trip. Of our total fleet wide capacity, 4% is used for systems operations, navigation and the media archives we brought with us. Another 1% is reserved for day-to-day operations, automation, user data, and fast transfer.

"That leave 95% free for stuff we add during our voyage. We used to have an idea of how long we'd be travelling, but that's out the window now. Let's put 50% aside for future data. The remaining 45% has to cover anything we want to unpack from the Orb, plus anything we build based on that data."

"That's still a lot of space" Kapri said, "should we just split it up per person?"

"That's a starting point," Ives replied, "but I think it should be a bit more developed than that. Imagine the Linguistics lab wants to do a large scale analysis of an alien language, that's going to take a big chunk of space, bigger than a personal ration could fit. So there needs to be some capacity set aside for larger projects being worked on by teams.

"At the same time, it's pretty important right now that everyone gets to keep exploring at their own pace. Accessibility is in the mission charter. If we carve the space up too rigidly and don't give people enough room to explore, we'll take much longer to find the real gems in this data."

Kapri nodded. "So maybe we split it. 25% for big projects, 20% for ad-hoc personal exploration."

"That's what I was thinking," said Ives. "And maybe a allocation vote system so that the fleet can democratically decide which projects to put into the project space."

"So how much working space does that leave for each of us?"

"Based on the average size of the packets we have to unpack, each person can look at maybe 2 or 3 'articles' at a time, depending on how much other personal data they've got stored. They'll have to recycle space often. Keep in mind, this is without our usual redundancy measures. Backups aren't feasible for most of this."

Kapri chuckled. "I bet data storage becomes one of the hottest research projects. Ok. Send me your charts, I'll put the recommendation together. Were you young enough to have a data plan, back on Earth?"

"No. My dad mentioned them, but I was always under the Gigamesh. You?"

"Yeah, as a teenager in Cairo. The warcore was taking up most of our digital resources, so we had household limits. I remember picking my songs for the week, checking my email, and downloading a couple shows, then doing everything I could to avoid pulling down extra data so mom could run her meditation classes. It wasn't great, but we made it work."

"Think the fleet will adapt so well?" Ives asked.

"Of course we will. We came out here ready to handle the challenges. Data organization may not be the challenge we expected, but it's better than having something trying to eat us. We just have to explain it well."

Ives smiled. "Well, that's why we have a Kapri."

Return to Story Index