News Archive
Last Loaded: 2018-06-07
49943/50k records

FAQ - News

October 27, 2018
05:53 PM Eastern
Sales Tax in the Age of TpT
So, for those of you unaware, TpT has tossed down the responsibility for collecting sales tax to you, the sellers. Yes, my wife is being impacted by this as well.

That means that, for the moment, I've set aside doing the ranking updates in order to build her a tool to calculate the states she's going to owe sales tax to, and to further calculate just what that total is on a state by state basis.

The good news for you, the sellers waiting patiently for the rank site to come back, is that when I'm done, I'm going to post it up on the web for you to use.

Here's the bad news...

Just from what I've read through so far, this is going to be a giant Charlie-Foxtrot (to borrow a military term) for everyone involved. On the same page of Colorado law I found three conflicting statements of whether my wife will owe sales tax or not. And that's just one state.

Added to that, TpT has given you, the sellers, only municipality data, and that data is self-reported, and often misspelled, mis-entered, or downright wrong -- one entry for my wife claims Texas is in the United States Virgin Islands.

I'm going to do my best, and when I'm done, I'm going to hand it over to you. But a word of warning, I'm a computer science guy, not a lawyer, not a tax accountant, and not a CPA. This will be a tool for estimating only, your mileage may vary, batteries not included, etc., etc.

I'm not sure if TpT is trying to chase away every seller they have, but I can't imagine a much more effective way of doing it than this fiasco.

I'll keep you posted.
September 30, 2018
11:56 PM Eastern
Just a "Little" Picture
After this weekend, I've gotten all the code for loading a seller's current state, and getting the list of new sellers set up and running.

The bad news: it's not exactly screaming fast. It took from Thursday morning until Sunday morning to just do the initial load of the sellers. That means if I really push it, I can load about 60,000 sellers in a day.

Which kind of points at the good news. There are 174,810 sellers loaded into the database right now -- sorry, it's not the one you have access to yet. Additionally, I've been trying to clean up the location data on the sellers, which means I had to geocode everyone's location to make sure it was accurate.

To put that in perspective, with 174,810 sellers, there are 24,360 unique locations that they reported. I haven't finished the full feedback loop yet, and there's a couple of things I screwed up with the Google Geocoding API (so a few people have the wrong geocoding, which I'll need to fix) -- but I got enough to feel good about building this:

That's a "heatmap" of all the sellers in the world with where they live. As you can see, there's quite a cluster in the U.S. and then a whole lot of small dots around the rest of the world, with a few bright spots in Europe and Australia.

I love insights like these, so now you have it. By the way, if Google wasn't so generous with their Map API, I would have been out $138.77 on the 26,000+ geocoding calls I made (not all of them succeeded but Google still "charges" for them.) That amount falls under the $200 of free calls Google lets you have every month. The best part? That counter is monthly and resets tomorrow, so if I have to fix a bunch of the geocodes, I'll still be able to sneak in under the "free" wire.

So, I hope that lets you know that things are back to some progress around here. To those who sent comments without a return email, let me just say, "Thank you for the encouragement!"

September 27, 2018
12:02 PM Eastern
So, it's been another month
It's been a month since my last update where I said I'd hoped to have more time, and I have had some, but most of it just ended up being used to tell me just how much TpT has changed everything.

Basically, they've removed pretty much every single one of the services and sources of data I was relying on to get data. About half-way through September, even the code that got the list of new sellers went away completely.

So, I've spent the last week or so re-starting from first principles.

On the plus side, that means that I've figured out how to get a reliable list of every single seller on TpT. On the minus side, I've lost nearly a month of data that I can never get back while figuring out how to get all this data.

Hopefully, this last method of pulling the data is one that TpT can't really eliminate, so at least I can get the current state of the sellers.

Again, on the minus side, it means loading a lot more data than I used to load, which means, if I want to be a "well-behaved-citizen" that I won't be able to provide daily updates on everyone's data.

But, I do have a way to get what people wanted, which is making participation voluntary, and making sure that you can't get rank for just anyone without either getting their permission, or owning the account.

I am still trying to figure out how to "reduce the friction" on you associating your account with you, and it's still looking like you'll need to have some kind of log-in to make that workable.

On the other hand, I'm hoping to offer more features, like I said, when I get all this working.

I haven't had as much time as I'd like, but I am getting data on all 174,500 sellers (as of this morning). So at least there's that...

August 22, 2018
02:47 PM Eastern
You Deserve an Update
So, I know I said back in July that I would have this site back online in a matter of days.

Yeah, that didn't happen.

This summer has proven extremely challenging. Apart from my actual job that already demands an insane amount of my time (along with the 2 hours of daily commuting time) we have had a series of challenges over the last couple of months.

Not all of them were bad. Brittany has gone back into teaching at a new charter school, but this meant pulling items out of 4 years of storage, and setting up a classroom (which was even harder because of construction issues -- think no A/C and a 93 degree classroom.)

Added to that, we had a massive, destructive hail storm back in June, and we are still dealing with the fallout of insurance, construction work, finding out the previous contractor didn't put on the Class-4 shingles he said he did, having to bring the roof up to code -- re-decking the entire roof of the house which isn't covered by insurance, and then having the water heater decide to die in the middle of it all.

All this added to moving people out of the house, and our daughter moving up to being a senior in High School.

To paraphrase Princess Bride, "I'm swamped."

The good news is that the system that builds the data I need for ranking has been running the whole time, and keeping the base data up to date.

The bad news is, I still haven't had time to actually finish the ranking engine, or build out the new website.

You all deserve better, but on the few times I've had the time, I haven't had the energy to work on it. I'm sorry for that.

Now that Brittany is back at work, and the kids are back in school, and the construction is nearly done (just one thing left to go,) I'm hoping that I will have some time to actually make progress, and I will try to keep all of you up-to-date as I go.

Once again, I'm sorry for the delay, but I promise I will make it there eventually.
June 22, 2018
08:42 PM Eastern
A few more numbers
So, looking a little deeper into the numbers.

  • 730 Responses
  • 701 Responses asked for the ranking site to continue (96.03%)
  • 29 Responses said no (3.97%)
  • 18 of the people who answered "no" would not change their opinion even if opt-outs were allowed (2.47%)
  • 707 Responses came from TpT Sellers (96.85%)
  • Being able to filter results by grade level and subject - Yes: 624 (85.48%), No: 97 (13.29%)
  • Log in accounts on the site: No: 463 (63.42%), Yes: 252 (34.52%)
  • Opting out of ranking: No: 403 (55.21%), Yes: 312 (42.74%)

Obviously some of these don't add up to 100%, because they did not require an answer.

What I came away with from this survey is as follows:

  • The rank site will continue. For now that means providing the historical, stored data that already exists in the database.
  • Over the next few weeks -- as I find time -- I will be evolving the parallel data collector, that has been running since last year to continue estimating ranks.
  • I will write a longer description of how rank will be determined moving forward.
  • Once I'm satisfied that the new algorithm is more than 99% accurate, I will re-run the estimates on all data since June 7th, to produce a set of full data.
  • I'll review this data, and then build the opt-out logic. Since people did not want log-in accounts, this may become "opt-in" logic. I'll explain that more later.
  • Opting out/opting in will not affect position in the ranking data.
  • One side effect of the changes is that the new site will index all 163,000+ sellers (as of now) on the TpT web site. Obviously most of these are lower ranked sellers. All 163,000+ sellers will be ranked.

Finally, the vast majority of those who left comments (over 200) were very positive about the site, vastly overwhelming the four negative comments.

This does not mean that I will not take those four negative reviews into account, as I would like to address the issues that they brought up that may be valid.

I hope this keeps everyone up-to-date.

June 16, 2018
12:36 AM Eastern
Results of the Poll
So, the results are in.

730 people responded to the poll, with 96% (701) of them asking me to continue the ranking web site.

As such, I have returned the web site to it's prior state, and your rank, as of early in the morning of the 7th of June, is once again available.

I had put off further work on the new site while this poll ran, so I will resume that work now. Please give me a little time to get the new site up and running.

I will be adding sorting on grade and subject with the new site, and I will also add the ability to opt-out of having your store name displayed.

Thank you for all of the supportive comments that were sent. I'll go into a little more detail at a later date, but it's late (just a little before midnight for me) and I have to drive about 150 miles tomorrow, so it's probably best if I went to bed now.

Thank you to everyone who answered the survey.

June 7, 2018
10:21 PM Eastern
The future is in your hands...
While I believe that I have been providing a useful service to the TpT community at large, today, when the site wasn't taken off-line, and with my news story that I planned to continue providing a rank of some sort, my wife has faced mounting criticism from many sellers on social media.

First of all, let me make it 100% clear that my wife is completely blameless in this matter. She hasn't even talked to me about my plans to continue to offer the website, or what form that would take. If there is anyone who deserves any grievances, it's me. Feel free to send them to me using the "Feedback" button at the bottom of the page.

Secondly, it is important for you, the end user of this system, to understand that I enjoy no special privilege of information or insight into the system at TpT. My plan to provide a future "ranking" was to train a machine-learning algorithm using the historical data so that it could "guess" at a rank. I would then return that as a "percentile" rather than a true ranking. This would be derived from the other public fields from the sellers, like number of votes, rating, number of products, etc. I only intended to do this because of the sheer number of feedback emails I've received about how inspiring it was for sellers to "move up" because it meant that they could see that they were doing something right.

Third, I make no money from this site. Ever. Let me make that clear. I have never made a penny. I lose money every month providing the bandwidth, the storage, and the electricity to run real, enterprise-grade servers for this site. I have never added advertising, even though Google Ads estimated I could make close to $4000 a month doing so. This is, and always has been a labor of love, and my way of giving back to the community that provided my wife the opportunity to make a career from providing curriculum to other teachers. This has been a giant "thank you" to that community.

So, with all of that said, I have currently disabled the search and rank functions of the web site, and in their place put up a poll for you to tell me what you want to see in the future. I want to keep giving back and providing the tools that TpT hasn't provided to make your life, as a seller on TpT easier.

But, I will abide by the results of the poll. If you tell me that you don't want a ranking site, then I will shut it down.

So, the future is in your hands.

June 6, 2018
01:39 PM Eastern
Your data is safe with us...
Just an FYI, this morning Cisco researchers announced that the Ubiquiti edge router was vulnerable to the VPNFilter attack. Have no fear, the router was already immune to the attack, because I'm not naive enough to use default passwords, or allow remote administration!

Still, within about 30 minutes, I had patched the edge router on the system to the latest firmware released a few days ago, and ensured that all the command and control networks for VPNFilter were blocked.

You may have noticed a few minutes of downtime as the edge router rebooted with the new security settings in place. Sorry about that.

So, your data remains safe here!

June 5, 2018
11:03 AM Eastern
All is not lost...
So, I've been working (and still will be working) on a new site to take over from this one after Friday's shut-off of ranking data.

I want to continue to offer something of value to the TpT community, and I think I can even provide you with something of a ranking as well, albeit a little less accurate.

I'm not done with the code yet, and I'm not sure I'll be done with it by Friday, but I really want this to continue to be a place that provides value to you, the TpT seller.

So, please come back on Saturday! Don't give up on the site. It may be a little rough around the edges at first -- hey, so was this site -- but I think it can grow into something very, very cool.

More to come.

April 11, 2018
11:19 PM Eastern
A Very Sad Day...
So, this morning TpT contacted me to give me a heads up (by a few hours) that they are going to be shutting off the ranking pages on the TpT site as of June 6th, 2018.

These pages are what define the ranks that I've been using to provide these pages for the last two years. Without them, I don't know what form this site will take.

As of now, all I can say is that things will change after June 6th. But for now, I don't know what form that change will take. I'm considering what I can do to continue to contribute to the TpT Selling community, but at the same time, the startup company I work for has also taken off, so "free time" is basically a misnomer at this point.

At the very least, you can rest assured that I will continue to keep the site up right through June 6th.

Watch this space to see if there's any news between now and then.

No matter what, it has been a pleasure to provide you, the TpT Sellers, with a service so many of you (over 5,000 unique visitors every month) have clearly found useful.

In other news, I did just change the site to fully HTTPS secure. Sigh.

March 23, 2018
06:06 PM Eastern
What was that loud bang?!?
That would be the sound of the wind knocking down my neighbor's tree... directly onto the power lines for the whole neighborhood.

Four hours and 13 minutes later, and we are back on-line. I wouldn't exactly say that the tree trimmers were, "rushing" to cut the thing up, but at least the power is back on now.

Sorry for the downtime, enjoy!
January 28, 2018
03:55 PM Eastern
A couple things...
So, this morning I applied all the accumulated security patches for the Meltdown and Spectre vulnerabilities. Hopefully very few people were affected, as the downtime was under 2 minutes.

On the other hand, someone pointed out the "Year-To-Date" graph wasn't working. Of course, this is because it's January. In January, when you are averaging across a month, with only one month in the year, you get one value...

It's kind of hard to draw a line graph with one value.


So, now, even when you choose "average" on YTD during January, you'll get a graph of the daily values. And, just to cap it off, if you choose YTD on January 1st, you'll get December 31st of the prior year plus January 1st.

So, even though I could have done nothing and the problem would have gone away in four days, I figured I might as well fix it.

This problem, by the way, was reported by someone using the feedback button. They didn't give me an email, so thanks to "Anonymous" you have a fix for the YTD graphs.

November 7, 2017
03:25 PM Eastern
Short downtime today.
Took about a five minute outage at about 1:20PM Mountain time to load some security updates on the main server and reboot it. Hopefully it didn't cause too much pain, as we were back up within about 2 minutes.

October 1, 2017
11:47 PM Eastern
Sometimes being a good citizen... kinda sucks.
So, today I got in a little work on the new system, trying to see if I could improve a few things. I went to the status page and what did I see but last night's record load times...

3 hours 20 minutes 30.788 seconds

Wow. Just wow. This was finishing up in under 30 minutes, even with all the new computing overhead I had added. It made me wonder what was going on.

So, I pulled up the Teachers Pay Teachers "robots.txt" file -- the set of instructions that tells the "bots" that crawl the web pages what rules to follow. And, low and behold, they've changed the default "crawl-delay" from 1 second to 20 seconds.

So, my new code being all friendly and a good "Net Citizen" honored that code. Meaning that it now takes about 12,000 seconds to crawl all 1000 pages. (There's a lot of reasons why it's not 20,000 seconds, but that gets all technical.)

Sigh, sometimes being a good, rule-following citizen of the web really sucks.
September 10, 2017
09:02 PM Eastern
Hey... What ya doin'?
So, you may be wondering what I'm working on. Well, I'm not completely ready to tell you all yet, partly because I haven't worked out all the kinks yet, but here's a little clue.

I know it doesn't mean much to the layperson, but to me it's all really, really neat.

Feel free to ask any questions you want answered through the Feedback item.
August 6, 2017
09:20 PM Eastern
Thunderstorms, the bane of my existence...
So, we just had another 20 minute outage, thanks to thunderstorms rolling through the area. Gotta love the weather in the second-most lightning hit city in North America.

Sorry about that.
July 30, 2017
11:57 PM Eastern
So, you may be wondering...
I really haven't made a lot of changes to the web page lately, and you may be asking yourself, "What exactly is he doing?" or even, "Has he given up on this site?"

The answer is a resounding, "No."

Okay, so what does that mean?

Well, in simplest terms, the work I've been doing has been behind the scenes. That means you don't get to see much of it happening, and not much changes on the front end of the system.

But much like the proverbial iceberg, 90% of the system is what you don't see.

For example, every night the system goes out to TpT and combs through all 1000 pages of ranking data. (For those that know what it means and care about that sort of stuff, I am a good net-citizen, and obey the "robots.txt" file for what files I can and cannot retrieve, and how fast I can pull pages.)

But, to be fair, I haven't really kept track of most of the data I could. I didn't do things like keep track of the new users being added to the system, or to the descriptions, or if you changed cities, etc.

So, under the covers, I've been completely re-writing the parser that goes out every night and pulls those top 50,000 spots. For those who have had issues where their ratings went away for a day, you know that the system isn't perfect, people move around while I'm loading the pages. There's even an FAQ entry about it.

The new system will still suffer from that issue on the load, but will then use the data from the prior day to intelligently "fill in the blanks" with the people who belong in those spots. So, there will always be a list of the top 50,000 users.

Finally, while the top 50,000 users get a daily update from the data, I've attempted to rank the remaining 45,553 users (as of now) into order by how many "votes" (rankings) they have. It's not a perfect system, because right now I don't have a way to update any user who isn't in the top 50,000 sellers, but I'm looking at what I'm allowed to search, and may even ask TpT if there's a way to get just an update on seller data in a compact form.

For now it's unlikely that anyone below 50,000 will move around in rank much, but I'm working on it.

So when will we see this?

Well, that's the little caveat to all this. Once I feel good about the new system, I need to convert all the web pages over to use it, and that will be a bit more work. And, my week of vacation ends today, so I won't exactly have gobs of free time to do it.

But rest assured, I have not forgotten about this site.

June 10, 2017
09:52 PM Eastern
Grumble... downtime...
So, I had to move the internet router from one plug to another. This should have taken a few seconds with almost no impact.

That was four hours ago. Four hours filled with many, many curse words and much frustration.

Everything is now working again, though I'm still not sure why everything went wrong in the first place.

Sorry for the downtime.
June 9, 2017
07:26 PM Eastern
Thanks a Million Plus!
Okay, this is late, but around the end of May, we reached our 1,000,000th visit to the web page.


That represents over 60,000 unique visitors, and we've served up over 600,000 rank graphs to those visitors.

During that time, I've transferred about 2 gigabytes of data every month for almost 20 months. That's about enough data to cover 40 truckloads of printed paper!

So, pat yourselves all on the back for making this site so successful! I hope you still think it's a great place to visit!

May 18, 2017
01:35 PM Eastern
Downtime for the new Edge Router
We just had another outage from CenturyLink, so I took the last 45 minutes or so and configured the new edge router. This meant downtime for everyone, but hopefully this will put an end to that issue in general.

Please send feedback if there's any problems with the website.

May 17, 2017
07:51 AM Eastern
Outage this morning (5/17)
This morning between 2:30AM Mountain Time and 6:30 AM mountain time, the system was off-line due to issues with the CenturyLink modem. I actually have a new edge-router to replace the modem and prevent this from happening in the future, but I haven't had a chance to configure it, as I've been putting in insane hours for work the last few days since it arrived.

I apologize for any inconvenience this has caused.
April 1, 2017
12:25 AM Eastern
Optimizing tables...
I'm currently (at 11:30PM local time) running an optimization on the rank tables. This can take up to an hour to run, and the response of the system may be pretty slow during this time.

Please bear with us while this goes on, as it should speed up the system overall.
March 24, 2017
11:51 AM Eastern
Outage on March 23rd
I have no idea why, but we had an outage on March 23rd that lasted nearly 2 hours. This was eventually solved by rebooting the router, but unfortunately, it chose to happen immediately after everyone had left the house. That was the largest reason that the outage lasted so long.

Sorry for any inconvenience, and, yes, I'm looking for ways to stop using that particular router. Unfortunately, that costs money, and time, and I don't have an excess amount of either of those things at the moment.

March 13, 2017
09:41 AM Eastern
A little bit of security upgrades...
This morning we had a little bit of downtime as I patched all the boxes to the latest security patches for the components.

Sorry for any inconvenience.
January 26, 2017
05:33 PM Eastern
We had an Internet outage here this afternoon and were down for about three hours.

Sorry for the inconvenience.
January 13, 2017
10:32 PM Eastern
Why does nobody tell me these things!
So, today I noticed that if you did "Year to Date" on the graph, with average values set, you got an error message. Oops.

This is because it's very difficult to draw a line diagram with only one point. So, the graphing library complained about it.

This is now fixed.

I wish someone had told me sooner. That's a perfect thing to use the "Feedback" button for!

December 22, 2016
09:09 AM Eastern
Database backups
As of this morning, there are 29,869,094 rank records in the database. Nothing particularly auspicious, except that I noticed the backup process hasn't been running in the last two months. Apparently the free utility I had been using wasn't capable of handling more than 25 million records.


So, as of this morning there will be a new backup process running each night at midnight GMT, which I think is about 7PM eastern time this time of year. It takes about 5 minutes to run, and responses might be a little bit slow during that period, so this is just a little FYI so you know ahead of time.

Because, you know, I'd hate to lose all 29,869,094 records. Of course tomorrow that will be about 29,919,050 records, and then 29,969,000 records, etc., etc.

October 24, 2016
03:04 PM Eastern
Thanks a lot, CenturyLink
So, if you hadn't noticed, we had several hours of downtime from last night until this morning. An amazing coincidence after I emailed CenturyLink asking them why their modem ports were opening a known attack port which was part of the denial of service attack on Dyn last Friday.

In any case, I had to walk my wife through resetting everything remotely, as I was 80 miles away from home when this happened.

Thanks a bunch, CenturyLink.

Update: From piecing together the logs, whatever CenturyLink did, they did it at 8:15 this morning (Mountain time zone.) We came back up pretty much exactly at noon, so we were down for 3 hours, and 45 minutes. Sorry again for the inconvenience.
October 4, 2016
03:24 PM Eastern
Time for some fall cleaning
I noticed that the response time on the ranks was getting pretty long (7+ seconds to pull back this year's records) and realized I hadn't "freshened up" the indexes in quite a while, so I kicked off an update of the indexes.

Problem is, it's going to take an hour or so to optimize all 20 million rows of data. During that time, the response will likely be even worse as the server chunks through all the data.

Sorry for any inconvenience.
September 21, 2016
12:47 AM Eastern
Hooray! Things are moving again!
It would appear the TpT has taken heed of the multitude of voices crying out in anguish and has repaired their ranking system.

Early this morning (East Coast Time at least) the system downloaded rankings from the site, and we once again have movement among the ranks.

Thanks to all of you who helped raise the awareness of the issue, we are back in the ranking business!

Thank you all!
September 13, 2016
07:22 AM Eastern
Something is rotten in Denmark... or New York, one of the two.
So, it appears that TpT is having problems with updating its rankings. We have not had anyone move since September 10th. (Okay, to be totally accurate a tiny handful of people moved one spot on the the 10th, but what I'm saying is generally true.)

This is not a problem on our end. The rankings we have exactly match the rankings on the TpT website. So, if you want to get this resolved, please post on the TpT Forum or contact the people at TpT to get them to fix their "Top Sellers" list as soon as possible.

September 12, 2016
06:57 PM Eastern
I love upgrades...
So, that explains why I wasn't getting any feedback on the new features. Turns out that when I upgraded the server and it also upgraded all the back-end software, it would no longer talk to the mail server to send the email I get when someone leaves feedback.

Don't fret, however. Being a paranoid coder, I also saved the feedback to the database, so I can still get it from there. In any case, the feedback is working again, so it shouldn't be a problem any more.

And this is why I love upgrading. All the little surprises that keep on giving.
September 12, 2016
11:01 AM Eastern
New News is New!
This morning I found out that, apparently, there are people out there that have never noticed the "News" items at the bottom of the page.

Well... really?

So, now when you first come on the page, if there's been a new news item since the last time you visited, you'll get a big red button that you can click on to find out the latest news prominently displayed at the top of the page.

Let's see you ignore that one.

Also, for those who never scroll down, you can also send me feedback at the bottom of the page, including suggestions for new features you'd like to see.

September 9, 2016
09:38 PM Eastern
Did you know you could request new features?
How many of you knew you could ask for new features? Apparently Lisa King and Amy (no last name given) did!

That's why you can now change the graph on the rank page from the full-sized, all-the-data-ever view down to a year, year-to-date, and monthly views. And you can see the year plus data averaged or in "raw data" form. That was Amy's suggestion, and I hope you like it.

Lisa King, however, asked for a way to download a month of ranks at once. Well, instead I added a "Download" button on the rank page that will give you back the exact same data that was used to generate the chart. So if you're looking at averaged data, you'll get a CSV file of averaged data. But, if you're looking at a month or a year of raw data, you'll get a CSV file with every single day of data back instead.

Hopefully this helps people out, and I'm always happy to try to help people out.

So toss a little love at Amy and Lisa, and enjoy the new features!

Also, since this is an all-new feature, let me know if there's anything wrong.

September 6, 2016
05:24 PM Eastern
Upgrade Time
Ubuntu 14.04.4 has gone out of support, and I need to upgrade the servers to 16.04 LTS.

What that means is, the servers will be getting upgraded. There may be some period of time where the server becomes non-responsive or acts a bit weird. Hopefully this won't be for a long period, but it's hard to say out of the box.

This will keep the servers up to date with security items, so it's very important that I do it as soon as possible to ensure that the system remains secure.

Sorry for any inconvenience while this happens.
August 16, 2016
02:18 PM Eastern
Half a million thanks!
Sometime late today (Friday, 8/12) or early tomorrow (Saturday, 8/13) we will serve out the half-millionth page from this site.

Half a million!

Thanks everyone!

Edit after the fact: We actually hit half a million visits late on 8/13/2016. Total visits by the end of the day were 500,087.

Thanks again!
August 10, 2016
08:07 PM Eastern
Yeah, down again... I'll let you guess why
If your guess was anything but, "Lightning," you need to go read the prior news items.
August 9, 2016
11:59 PM Eastern
Colorado Springs is not the lightning capital of the US
No, that dubious honor goes to Tampa, Florida. We are, however, in second place.

Once again, tonight, we had another lightning strike take out the power, and some glitches in coming back up. (And if you think that's bad, you should deal with my daughter when the Olympic Gymnastics get cut off in mid-stream.)

Anyway, I once again had to run the rank update manually, but everything should be back to normal again.

Now, if I could just get the lightning to stop.
August 8, 2016
01:01 AM Eastern
Neither rain, nor... oh wait, never mind.
So, we had a very large thunderstorm come through, which resulted in a very long period of going back to living like the pioneers. Yes, that's right, we had no access to Facebook.

Worse, we had no power, so the servers shut down. When the power did come back, they didn't quite all start back up the way they're supposed to, so it took a little work to get everything going again. Unfortunately it happened about 5 minutes before the data load should have run, so I'm having to kick it off a bit late this morning, manually.

Also, we were off-line for about 3 hours. So that whole "rain, nor sleet, nor snow" thing? Well, so far we're good on sleet. Sigh.
Sorry for the inconvenience.
July 27, 2016
06:21 PM Eastern
Oh, one more thing...
You can now search for countries from the main search page. Because, hey, why not?

July 27, 2016
04:11 PM Eastern
City, State, and Country lists are now... prettier
I added the "Next" and "Previous" to the city and state pages, and those pages now automatically highlight your position in green.

July 25, 2016
09:42 AM Eastern
Runaway! (Let me know if there's any problems today)
Last night I had a run-away process use up all the disk space on the server that pulls all the ranks. I don't think this affected the pull of the data (there are 49,999 records today) and it shouldn't have broken anything, but please let me know if you have any problems with the website today.

The file system has been cleaned up and this shouldn't happen again (hopefully.)
July 22, 2016
02:07 AM Eastern
So, while I was stuck at work after hours...
Tonight, well, last night, actually at this point, I got stuck at work. I could have driven home -- an hour drive -- just barely, but then I would have had to get back on-line to work from home because we had a software load tonight. (If you're really interested, I'll explain more below.)

But, I had an hour or so to kill, so it occurred to me, I could use that to do something.

And, what came to mind was a question Brittany was asked on TpT the other day. Basically, a lady asked her how to find her rank by country on the web site.

Now, as I'm sure a lot of you know, it's already visible on the rank page, but unlike the state or city ranks, you can't click on it to see a list of the whole country.

Well, now you can.

And, as an added bonus, you can now scroll through the country list using the "Previous" and "Next" items at the bottom of the list.

You'll automatically be placed on the page where you show up for the country. I imagine that when I have another opportunity like this, I'll do the same thing for the City and State lists.


For the curious:

I work as a software architect and development manager for a software company in Denver, but live in Colorado Springs. About once a month we load the new version of the software to servers in Denver and Phoenix and North Carolina. These loads typically start late at night, and I work on them remotely by logging in from home.

Tonight, there was a "crisis" at 5PM, and I got stuck at work until nearly 9PM. The load starts at 10PM, and while I could have gotten home (barely) in time to work there, I figured I'd just stay. Ahh, the glamorous life of the software engineer.

Still, I added this, so at least some good came out of it.

June 29, 2016
09:43 AM Eastern
Somebody moved...
So, I got this bit of feedback this morning:

"All the ranks I looked at today have stayed the same except all have a +1 USA rank." - PrintPath

"Oh no," I thought, "there's another problem."

So I checked, and it does seem like a lot of people have a rank of +1 in the U.S. And why did this happen?

Well, the simple answer is that, now that I'm going a lot slower pulling the ranks, I'm a lot more vulnerable to people "moving" while I'm grabbing pages. TpT is a real-time system, meaning even as I'm grabbing the ranks at 1:30AM Eastern Daylight Time (5:30 AM GMT) there are still sales going on. And since a sale affects someone's rank, they can move within the ranking list. That happened last night, when whoever was at rank 851 (I went back and checked, it was "The 3am Teacher") made a sale sometime between 1:30:10.802 AM, and 1:30:11.732 AM. This caused them to pass the person at spot 850 between when I loaded page 20 and when I loaded page 21 of the ranks.

Here's what that looks like in the logs.

2016-06-29 05:30:10,802 INFO  [ 81] WorkerThread - Loading:
2016-06-29 05:30:11,732 INFO  [ 81] WorkerThread - Loading:
2016-06-29 05:30:12,045 ERROR [ 147] WorkerThread - Found an existing seller for seller_id=717532 -- Sailing into Second at rank 850 -- new rank: 851

So, the upshot of all this, is that "Sailing into Second" moved from 850 to 851, and we skipped over "The 3am Teacher", who moved up to spot 850. This means the records in the system now look like:

490230Adaptive Tasks by Dianne Matthews848United States - Missouri - Ozark
1360103Bond with James849United States - Texas
717532Sailing into Second851United States - California
1156646Teaching With Balls852United States - Oklahoma

As you can see, there is no entry for "The 3am Teacher" today (850 is missing) -- this is one of the disadvantages of the slower loading speed. People move.

Since the US, state, and city ranks are based on literally counting the people ahead of you in that location, that means everyone bumps up a spot today.

In fact, if you look up "The 3am Teacher" today, you will see that she is ranked at 851. This is because I cleverly coded around this problem, and pick up her rank from yesterday, rather than from today. (Okay, "clever" is a computer programming term for, "I cheated and solved an easy problem rather than solving the hard one."). If you look up "Sailing into Second" they are also at rank 851 today. That's because their ranking is today's, which is what you'd expect.

In reality, this happens a lot during the load. Last night, in the 458.3 seconds it took to load all 1000 pages, there were 41 people who moved during the load, meaning there are 41 missing entries. That's why, up in the top corner, there's a counter that shows how many entries were actually retrieved of the 50,000 possible. Today, it's 49,959 entries, because of those 41 people who moved around.

Anyway, that's why everyone is one spot higher in the U.S. today, if your rank is over 850, even if you didn't move in the world rankings.

In fact there's another gap at 3350, and another one at 4651, and another one at 9701, and so on...

I'm doing the best I can here, people. You'd all make it a whole lot easier if you'd just stop moving around! :-)

Thanks for the feedback, and thanks for using the website!
June 28, 2016
09:55 AM Eastern
Tuning time...
It looks like I can now only run 4 simultaneous skimmers before I start getting errors from TpT, so I'll be reducing to that number.

I'm not sure if this was an intentional change made by TpT or not, but it's definitely a lot slower than I used to be able to grab pages. This means that the load times will now be in about the 10 minute range every morning.

Please let me know if there's any more problems tomorrow (the 29th) or afterward.

I'm not sure what I can do about the somewhat "wonky" data left over from the 27th. Hopefully I can do something about it to patch the 4000 or so missing records, but I'm not sure if the better choice is to just do nothing, as the bad city/state/country rank movements will clear themselves up tomorrow when the new data loads.

Again, sorry for the inconvenience, and thanks to those that alerted me to this problem!
June 28, 2016
09:24 AM Eastern
Root causes
So, it looks like TpT has put some pretty heavy throttling on how fast I'm allowed to load the pages. Before I was getting them as fast as possible because the shorter the period of time, the less people would "move around" within the rankings.

So, I would grab all 1000 pages in about 3-5 minutes, using 100 separate page-loading threads.

This appears, now, to be a big no-no with TpT, as when I start the loader, it almost immediately starts getting clamped down for excessive speed. Running it single-threaded means I can get all the pages (and it's currently re-loading all of the data for today) but it's not exactly screaming fast, doing about 1 page every 1.4 seconds, meaning the 1000 page update is going to take nearly half an hour.

I'm going to play around with the algorithm a bit to see if I can get it to go any faster, but for now, I'm going to leave it at "slow" and make sure the loads work, rather than get the speed back up.

Sorry for the inconvenience that this has caused for anyone.
June 28, 2016
09:13 AM Eastern
See if you can tell when something changed
-rw-rw-r--  1 jnaujok jnaujok    163503 Jun 12 05:30 FindPeople.log.2016-06-12
-rw-rw-r--  1 jnaujok jnaujok    164862 Jun 13 05:30 FindPeople.log.2016-06-13
-rw-rw-r--  1 jnaujok jnaujok    164008 Jun 14 05:30 FindPeople.log.2016-06-14
-rw-rw-r--  1 jnaujok jnaujok    163687 Jun 15 05:30 FindPeople.log.2016-06-15
-rw-rw-r--  1 jnaujok jnaujok    163672 Jun 16 05:30 FindPeople.log.2016-06-16
-rw-rw-r--  1 jnaujok jnaujok    163503 Jun 17 05:30 FindPeople.log.2016-06-17
-rw-rw-r--  1 jnaujok jnaujok 329979844 Jun 18 06:47 FindPeople.log.2016-06-18
-rw-rw-r--  1 jnaujok jnaujok 296816808 Jun 19 06:21 FindPeople.log.2016-06-19
-rw-rw-r--  1 jnaujok jnaujok 289385348 Jun 20 06:18 FindPeople.log.2016-06-20
-rw-rw-r--  1 jnaujok jnaujok 275255482 Jun 21 06:38 FindPeople.log.2016-06-21
-rw-rw-r--  1 jnaujok jnaujok 255476868 Jun 22 06:16 FindPeople.log.2016-06-22
-rw-rw-r--  1 jnaujok jnaujok 315458390 Jun 23 06:28 FindPeople.log.2016-06-23
-rw-rw-r--  1 jnaujok jnaujok 237089036 Jun 24 06:14 FindPeople.log.2016-06-24
-rw-rw-r--  1 jnaujok jnaujok 696975560 Jun 25 09:01 FindPeople.log.2016-06-25
-rw-rw-r--  1 jnaujok jnaujok 545256492 Jun 26 07:27 FindPeople.log.2016-06-26
-rw-rw-r--  1 jnaujok jnaujok 317478444 Jun 27 06:22 FindPeople.log.2016-06-27
June 28, 2016
08:25 AM Eastern
So, there's a bit of an issue...
Teacher's Pay Teachers' rank site apparently was offline, and that resulted in the "skimmer" for ranks failing to work for the last two days.

This is leaving the web page in a bad state. I'm working to find the root cause of the problem, and to code a fix around it. The rank pages are back up on TpT, but for some reason, the skimmer process is still getting "502 Errors".

Because the skimmer retries errors, the system hasn't actually gone into an "error state" and failed the load from yesterday morning. This blocked the load from this morning starting, and it's just kind of snowballing from there. That's been giving some rather odd results for some people, as the web pages sort of assume (yes, I know what that word means) that there's always at least some data for today.

I'll be working today to see what I can do to clean up this mess. In the meantime, it looks like the TpT rank pages are back on-line.

Please bear with me as I work on this problem.
June 19, 2016
06:51 PM Eastern
Sweatin' to the... no wait.
Thanks to the 100+ degree temperatures here today, the database server overheated. Following some hasty re-arrangement of cooling fans, the server should be back on-line now.

Sorry for any down-time.
May 20, 2016
03:56 PM Eastern
Call me Ishamael... The CenturyLink Saga is Finally Over.
After 8 months, the cable my neighbors cut while putting in a new fence has finally been replaced from end to end.

We were down for about two hours or so as two men literally hand-dug a trench for 150 feet to lay the new cable. Seriously. No trencher tool, no automatic anything, just two shovels.

Admittedly they only dug about four inches deep, and the whole place is beach sand, so it wasn't too challenging, but it still boggles the mind that they didn't have some little gas powered trencher to do this for them.

Anyway, this will hopefully end the saga of issues I've had chasing this cable cut for 8 months. At least it wasn't snowing today...
May 20, 2016
10:02 AM Eastern
In Theory, CenturyLink Will Fix the Phone Line Today...
Sometime today (gotta love contractors) CenturyLink will be showing up to fix the phone line that got cut 8 months ago.

Yes, 8 months.

So, when that happens there will be an outage for an indeterminate amount of time. Hopefully this will be short, as I will try to convince them to only cut the old line when the new one is ready to go in.

Crossed fingers...
May 10, 2016
07:44 AM Eastern
Network outage
Apparently we lost the network for almost 5 hours this morning (4 hours, 47 minutes). I have no idea why, as the incoming connection was good and solid, but it just wouldn't give me link-up for the actual Internet.

After rebooting the router, the network did come back on-line.

Sorry for the inconvenience.
May 9, 2016
01:46 AM Eastern
Over 360,000 Pages Served
I just happened to glance at the counters and caught us at exactly our 360,000th page visit.

Over 22,500 unique visitors have come to the page, and over 234,000 rank pages have been served.

Thanks again for everyone coming to visit!
April 29, 2016
11:00 AM Eastern
CenturyLink! From Hell's heart I stab at thee!
So, it's snowing...

Which means the guy took one look at the cable break and said, "Nah, we can't fix that today. Let me schedule a time a month from now..."

I'm starting to know how Captain Ahab felt...
April 28, 2016
09:19 PM Eastern
CenturyLink! You task me!
So, tonight we had a city-wide outage of Internet service on CenturyLink. We were down for about two and half hours. I think we're back up now.

But, tomorrow morning sometime between 8AM mountain time and 1PM mountain time, we'll be back down again.

"But why?" you may ask.

Well, because finally, 7 months after my fence-pole digging neighbor sliced the phone cable, and I went out and hand patched it, the CenturyLink technicians are finally going to show up to actually repair it.

Thus, the quote from Melville.
April 11, 2016
04:35 PM Eastern
Swedish... really?
For some reason when the database came back up, it was set to "Swedish" as its default.

Seriously... Swedish.

Anyway, I'm reconverting the tables to use UTF/Unicode (utf8_unicode_ci for the SQL savvy). That's going to take a few minutes, and you may get strange results while it's going on.

It only took about 20 minutes, the conversion is done. Hopefully everything is back to normal-looking. Bork, bork, bork.
April 9, 2016
08:17 PM Eastern
Well, that took longer than I planned on...
First of all, I'm sorry for the downtime we had today. I did warn you it was coming, but I was planning on 2-3 hours, not 12.

Everything should be working now and generally back to where it was. Hopefully the response times are a little bit quicker as well, as both the web server and the database server are now running on a clean install on SSD drives.

Please let me know if there's any issues in the feedback below.

By the way, if you left feedback between any time before about 6PM eastern, that feedback got lost when I had to restore the database to where it was this morning.

Once again, sorry for the inconvenience, but hopefully this is the groundwork for getting to a much more robust and better backed-up system.
April 9, 2016
06:52 PM Eastern
Almost done...
After quite a few headaches and missteps, the servers are almost done.

There will be a few more minutes of downtime when I shut them down and move them back to their server rack.
April 6, 2016
01:16 PM Eastern
Trying again... Downtime this Saturday, April 9th.
Now that I have the parts in hand, I'm hoping to do the updates and upgrades this Saturday, April 9th.

The site may be up and down during the day as I attempt to make the servers better, faster, stronger. Or maybe I'm thinking of the Olympics.

Sorry for any inconvenience this causes.
April 4, 2016
06:38 PM Eastern
The Graphs are Broken! The Graphs are Broken! No, wait, they're fixed.
So, I checked the site today and noticed that the graphs were suddenly very skewed for the April 2015 values.

Now, some of you have asked about the graph, so let me explain how it works, so I can explain why it was broken.

The graph shows the average rank you had for that month. There's two reasons I did this. Number one, plotting 350 data points takes a lot longer than plotting 11 or 12 data points, and I do need to keep the load on the server reasonable. Number two, back in October of 2015, TpT was having issues with its rankings pages, and this was introducing "random" values (+/-100 places) into the data. By using the average, the randomness vanishes into the whole and makes for a much better looking graph.

So, how did I do that? Well if you know SQL, I did "select average(rank) ... group by month(rank_date)".

For those of you who don't speak database-ese, that means, "give me the average rank for each bucket of values, where the buckets are grouped by month."

See the problem?

I forgot to include the year. Which worked fine through March. But the oldest data I have goes back to April of last year, which means, starting on Friday, I was averaging April of 2015 with April of 2016.


That's been fixed, and the graphs are back to normal.

It also means that on April 20th, there will be a full year of data for every person in the top 50,000 sellers on TpT. For those of you doing the math, that's 18,300,000 records. (50,000 * 366) [366 because we had a February 29th in there.]

That's a whole lot of data. Believe it or not, I'm actually working on a way to collect even more data. Hopefully you'll all like it when I get it done.

March 26, 2016
12:45 PM Eastern
No downtime today...
Apparently UPS decided that 50 degrees and sunny was too dangerous to ship my parts from Denver to Colorado Springs, so there will be no downtime today, as I won't get the parts until Monday now.

I will update when to expect downtime when I know.
March 23, 2016
02:08 PM Eastern
Planned Downtime on Saturday 3/26
On Saturday, I'm going to be adding some new parts to the hardware on the servers, namely some remote-administration cards. During this time, the servers will be down for about an hour.

Also, the web server has been sending warnings that one of the disks is in "pre-fail" status, so I will also be replacing one the hard drives with an SSD drive, which will also hopefully make the system a little bit faster.

Secondly, I'll be changing the database drive to an SSD drive as well, which should also give a much higher response rate, as well as providing a little more backup support for the data as well.

Sorry for any inconvenience this causes.
March 6, 2016
04:33 PM Eastern
Power failures, and bad updates, oh my!
Early this morning we had a power failure, which resulted in the servers exceeding the UPS' backup capability. Apparently when the power did come back on, it did it through a series of "flickers" that left the servers in an untenable state. That, and one of the servers had apparently been in the middle of an update when the power went out, so I had to back out those changes, and then restore them.

I believe that has now all been remedied, so the site should be back up and fully available.

Please let me know through the Feedback link if you encounter any other problems.

We apologize for the inconvenience. Thanks for your support.
February 27, 2016
09:13 PM Eastern
Who needs headaches... when you have servers?
So, today I did some database updates, which led me to install security updates, which led to the web server not restarting which led to me pulling the server rack out so I could connect testing equipment.

After I got everything all back together, I ran some tests, and the server was back up. However, I didn't realize I had accidentally pulled the database server's network cable loose.

Normally I would have caught that, but as a precaution, last week I added a second network to each server, so the machine happily hummed along, saying it had network connectivity.

The problem is, the web server only knows how to talk to the database server on the network connection that got pulled out. So, while all the servers happily reported as healthy, the web server couldn't actually reach the database server. It wasn't until an update process complained that I realized something was wrong.

Sorry for the inconvenience.

Let me know if anything isn't working as expected.
February 27, 2016
01:27 PM Eastern
Database Maintenance
Today (Feb 27th) I'm doing some database maintenance to try to keep the responses quick and stable. With nearly 15,000,000 records in the rank table, the database is needing a little "spring cleaning" to keep it chugging along.

You might notice a little lag or slowdown in the response while this is going on but it should be done soon (Hopefully by 4PM Eastern time).

January 31, 2016
05:33 PM Eastern
I was installing a modified email package on the server, and apparently it really didn't like the database connector that the rank site was using to talk to the database.

It's been fixed now. Sorry for the inconvenience.
January 6, 2016
01:28 AM Eastern
Over 100,000 Served!
I just checked our page counter, and we have served out over 100,000 rankings since we first put this site up back in August!

According to our counters, in 2015, we served out two thirds of a million pages to nearly twelve thousand unique visitors! Those pages totaled nearly 5 gigabytes of data. That's enough to fill 9 CDs!

Thanks for all the visits and we're so happy to have provided a service that people find useful!
December 24, 2015
10:14 PM Eastern
Merry Christmas and Happy Holidays!
From Brittany and Jeff here at the TPT Rank Search site, we wish you all a Merry Christmas, Happy Holidays, and a Happy New Year!
December 7, 2015
02:54 PM Eastern
With humblest gratitude...
I would like to thank all of you who have taken the time to send such overwhelming praise via the feedback form. It is truly humbling and inspiring to know that we have helped so many good people.

Thank you so much!
December 1, 2015
12:23 AM Eastern
Hopefully More Mobile Friendly
I got some feedback that the pop-up help was living up to it's name, and popping-up... but not popping back down, so the help was covering up the list of ranks on mobile devices with smaller screens.


I've re-organized the web page a bit and the help pop-ups should now disappear after a short time-out, leaving you free to check your rank on a phone.

Hope that's better!
November 30, 2015
06:22 PM Eastern
A Little Bit of Bad Data...
Looks like last night's run is missing 230 records -- a few of you may have problems getting your rank today.

Normally we've been seeing 2-10 duplicates -- this happens when someone was at, say 1050, the last entry on page 21, and then when I load page 22, they are at 1051. Last night, there were 80 of those duplicates.

Hopefully this is a one-time glitch, with a lot of people in motion because of the sale that kicked off today.

It also looks like 3 pages just didn't load at all. I'll have to look into why the process didn't just re-load those pages.

Update: It looks like two of the pages missed are pretty early in the set of pages, because a lot of people have "moved up" quite a few spots in the US/State rankings. I've updated the loader code to handle this better, and it should fix itself tomorrow. Unfortunately, I can't go back and re-load the pages as they were this morning.

Sorry for the inconvenience.
November 30, 2015
05:28 PM Eastern
Upside-Down is Right-Way Up
For those who commented that it's hard to look at a graph going down as a good thing, I've reversed the graph for ranking (and made it a bit more... shiny... as well.)

November 27, 2015
11:42 PM Eastern
Now with pictures!
Okay, I've finished up the face-lift, and I added one extra finishing touch -- a graph on the rank page.

Thanks to the history kept in the database, we can provide your average rank for every month since April when we started pulling down the rank data. So, now you can see it in lovely graphical form as well.

Thanks for all of you who visit here to get your rank! It's great to know that we're doing something to help the community.

November 26, 2015
07:36 PM Eastern
Things look different around here...
Since I've had a few days off for the holiday, I decided that it was really time for the site to get a face-lift.

The site was originally put together in about an hour and a half, and then got a makeover a few days later, but I really haven't had a lot of time since then to "touch it up".

So, hopefully you like the way things look. I've also added a new "Feedback" section just in case you have any comments or would like to see something else changed, or if you've found any other problems.

November 20, 2015
02:00 AM Eastern
Good News!
It looks like your requests have had an effect, and TpT's ranking system isn't glitching any more! For the past seven days it looks like there haven't been any ranks jumping up and down. I'm still keeping an eye on it, but it's looking a lot better. Thanks for all the help!
Optionally enter your full name
Optionally enter your email
This system is being provided free of charge by The Colorado Classroom, LLC.

This system uses only publicly available data from the TPT website. None of this data is privileged or private in any way.

This site has no affiliation with Teachers Pay Teachers, nor is any claim made on any of their intellectual property, copyrights, or other information