Canada's 'Random' Immigration Lottery Uses Microsoft Excel, Which Isn't Actually Random (gizmodo.com) 224
An anonymous reader writes: Last year, Canada introduced a new lottery system used to extend permanent-resident status to the parents and grandparents of Canadian citizens. The process was designed to randomly select applicants in order to make the process fairer than the old first-come, first-served system. There's just one problem: the software used to run the lottery isn't actually random. The Globe and Mail reported the Immigration, Refugees and Citizenship Canada (IRCC) uses Microsoft Excel to run the immigration lottery to select 10,000 people for permanent resident status from a field of about 100,000 applications received each year. Experts warned that the random number generating function in Excel isn't actually random and may put some applicants at a disadvantage.
First, it's best to understand just how the lottery system works. An Access to Information request filed by The Globe and Mail shows that IRCC inputs the application number for every person entering the lottery into Excel, then assigns them a random number to each using a variation of the program's RAND command. They then sort the list from smallest to largest based on the random number assigned and take the first 10,000 applications with the lowest numbers. The system puts a lot of faith in Excel's random function, which it might not deserve. According to Universite de Montreal computer science professor Pierre L'Ecuyer, Excel is "very bad" at generating random numbers because it relies on an old generator that is out of date. He also warned that Excel doesn't pass statistical tests and is less random than it appears, which means some people in the lottery may actually have a lower chance of being selected than others.
First, it's best to understand just how the lottery system works. An Access to Information request filed by The Globe and Mail shows that IRCC inputs the application number for every person entering the lottery into Excel, then assigns them a random number to each using a variation of the program's RAND command. They then sort the list from smallest to largest based on the random number assigned and take the first 10,000 applications with the lowest numbers. The system puts a lot of faith in Excel's random function, which it might not deserve. According to Universite de Montreal computer science professor Pierre L'Ecuyer, Excel is "very bad" at generating random numbers because it relies on an old generator that is out of date. He also warned that Excel doesn't pass statistical tests and is less random than it appears, which means some people in the lottery may actually have a lower chance of being selected than others.
So... (Score:2, Insightful)
Why not just accept all the immigrants who show up? That's what they tell the US to do, right?
Re: (Score:2, Insightful)
Who is the "they" you speak of? I can think offhand of anybody who says "accept all the immigrants who show up."
This is technically the libertarian position, true (libertarians consider borders to be government red tape restricting people's natural freedom to go wherever they want), but most libertarians are smart enough to mostly work on less hot-button issues, like freedom of guns and drugs, instead of freedom of borders.)
Re:So... (Score:4, Interesting)
There is explicit, and there is implicit. There is a whole slew of people who think that if you can sneak across the border, that you are entitled to live in America unmolested for any reason because "oppression". I would consider this implicit "accept all immigrants who show up" stance. It is largely why Donald Trump wants a wall built, and none of these other people want a wall. Again, building the case for implicit "accept all". And then there is the Sanctuary Cities. And and and ...
On the Explicit category, there is me. I explicitly state that ANYONE that shows up at the border, and checks in (documented) should be allowed in*. Yes, I am a libertarian. I also want to remove the stupid policies and fake fighting over the (D) party who loves illegal aliens, and the (R) party who loves cheap labor and remove those policies from existence.
I realize that I am in the minority for stated opinions, but the reality is, there are enough people who already "want" it in some way or fashion, I'm just honest about it.
*Other conditions apply, but for simplicity sake, this is the basis for my policy.
Re: (Score:2)
On the Explicit category, there is me. I explicitly state that ANYONE that shows up at the border, and checks in (documented) should be allowed in*.
OK, I'm curious now. Why do you think documents are important? Why do you think a government should be given the power to issue or not issue documents to allow a person to cross borders?
Re: (Score:2)
You'd rather revert to serfdom and be unable to leave the land you were born on?
Re: (Score:2)
If One Billion people show up, we would survive. We would figure it out, and there would be a boon for Real Estate, and cheap labor. It would shake the status quo to its core, and perhaps break it in ways neither of us could fathom today.
If One Billion people show up, other countries would suddenly struggle to have any economy. People, are both a drain on resources and the most valuable of all resources.
If One Billion people show up, they all wouldn't show up at once. It would take a number of mechanisms fo
Re:So... (Score:5, Informative)
Most libertarians probably wouldn't care about immigration if the U.S. had no welfare state as was historically the case throughout most of the country's history. Once you got off the boat you were on your own, but plenty of people were perfectly happy with that arrangement. This generally worked quite well with the homestead system since anyone who wanted to move west could do so and a large number of immigrants did. You had the freedom to make something of yourself and if you couldn't that was your own problem.
However, the reality in modern America is that the government is expected to care for anyone who shows up and pay for their children's education and healthcare. That necessarily means taxation, which libertarians are generally loath to agree to outside of the limited government functions that they're in favor of, so they'll tend to be less in favor of immigration with those kinds of strings attached to it. If everyone immigrating from Mexico were a doctor, engineer, or otherwise highly skilled such that they'd pay more into the system than they get out of it, I suspect there'd be no disagreement with letting in as many people who fall into that category either.
confusing libertarians with anarchists [Re:So...] (Score:2)
I think you're confusing libertarians with anarchists.
That wouldn't be surprising. Libertarians often confuse libertarians with anarchists.
LIbertarian's believe in whatever's convenient (Score:4, Interesting)
Even Ayn Rand [goodreads.com] took social security in her old age. Though to her credit she had to be convinced to take it rather than die in the street. Her writings weren't profitable until the Republicans decided they needed an intellectual
My experience with Libertarians is they're folks who never grew out of that phase in your teenage life where you really, really hated being told what to do. You know the one. It's when you're just starting to realize how capable you are, when you're at your peak of learning capacity and you're figuring things out faster than the adults. And you really are (teenage brains work that way).
What I find especially maddening is the libertarians who rail against coastal elites and SJW and are perfectly OK with billionaires having unlimited wealth because, hey, they earned it by virtue of having it. Never mind the fact that money is power and you can't be free in a world with that much wealth inequality. After all, you're not free if somebody controls your access to food, shelter, healthcare, education and transportation (the latter needed to access the former). You're one week's food, one winter's cold or one pill away from slavery. True freedom only arrives when everybody has their needs cared for not because they can threaten or cajole people into getting it but because they're humans, and humans have a right to those things.
Re: (Score:2)
Who is the "they" you speak of?
Bernie Horowitz.
Re:So... (Score:4, Interesting)
Who is the "they" you speak of? I can think offhand of anybody who says "accept all the immigrants who show up."
This is technically the libertarian position, true (libertarians consider borders to be government red tape restricting people's natural freedom to go wherever they want), but most libertarians are smart enough to mostly work on less hot-button issues, like freedom of guns and drugs, instead of freedom of borders.)
I've also noticed that most libertarians aren't really libertarians. Most people I've met who self-describe as Libertarian tend to be Republicans who are pissed off at the Republican party and call themselves libertarian as a protest. When I ask them do you believe "x, y, and z" (insert libertarian positions) the answer is always, "well no... but I think we need less government". Libertarian is not republicanism minus racism and Libertarians are not just republicans that smoke pot. That's not what libertarian means.
Re: (Score:3)
>"That's not what libertarian means."
Nothing is really an absolute position, regardless of party. But, in a nut-shell, Libertarians believe in less government. That equates to things like:
1) Fewer regulations
2) Fewer laws
3) Less taxes
4) Less government spending
5) More personal freedom
6) More personal responsibility
7) More local control (less Federal)
Notice I didn't say "no" or "none" or "all" in the above. Just guiding principles. For some reason, on Slashdot, for many people, the word "Libertarian"
Re: (Score:2)
Words like "fewer", "less" and "more" are useless, because it's not clear how much, and compared to what baseline. Those words give you a nice fuzzy feeling without actually committing to a debatable position.
For example: "Less government spending". How much less, and where exactly will the money come from ?
Re: (Score:2)
>"Words like "fewer", "less" and "more" are useless, because it's not clear how much, and compared to what baseline."
They are not useless. We are discussing 3 parties, so it can easily mean less or more than the other two.
>"For example: "Less government spending". How much less,
Considerably less. Most I know wouldn't even blink at saying half. But there is no exact amount, and change takes time, anyway.
>"and where exactly will the money come from ?"
What money? The money we aren't spending? From
Re: (Score:2)
What money? The money we aren't spending?
The money from this pie chart shows where the governments spends the money:
https://www.usgovernmentspendi... [usgovernmentspending.com]
When you cut the government spending in half, which of the sections of the pie chart will be reduced/removed ?
Re: (Score:2)
>"That's not what libertarian means."
Nothing is really an absolute position, regardless of party. But, in a nut-shell, Libertarians believe in less government. That equates to things like:
In my younger days I WAS a libertarian. Growing up I've abandoned that- I think it sounds better in theory then it is in practice, so yes, I know the basics of Libertarianism. (I still believe in completely free movement of goods AND people, for example - but I think it has to work in all directions to work; until other countries open their borders it won't work for us to do so alone).
As a heads up, I now live in bible belt South Carolina so my experiences may not be the same as most places. Most peopl
Re: (Score:2)
Who is the "they" you speak of? I can think offhand of anybody who says "accept all the immigrants who show up."
We know who "he" is (op). A trumpist, most likely a foreigner, possibly Russian. Hanging out on social media to seed discord by being an idiot. Typical trumpist. No real person would do it for free.
Re:So... (Score:4, Informative)
That's effectively the position of Democrats.
Is it? Can you quote a single Democrat saying this? Ever?
No, I didn't think so.
This is technically the libertarian position
Libertarianism also favors small government, freedom of association and private property rights, all of which are severely limited; those are necessary prerequisites for libertarian style open borders.
Yep. Libertarians are pretty good at avoiding mentioning their position on the hot-button issue of immigration. Guns and drugs, that's what they want to talk about.
Re: (Score:2, Insightful)
"Think of it as evolution in action," that's their catch-phrase.
--say, we may need those immigrants after all!
Re:So... (Score:4, Insightful)
These days the main libertarian issue is to get rid of Obamacare and let people simply die when they get sick.
"Think of it as evolution in action," that's their catch-phrase.
--say, we may need those immigrants after all!
It worse than that "I am okay, so nobody needs it", until things are not so rosey and then it is "what happened to the free market that was meant down health care prices?"
Re: (Score:2)
Re: (Score:2)
try again.. [thinkprogress.org]
Obama, the-deporter in-chief (Score:2, Informative)
That's effectively the position of Democrats.
Didn't seem to be Obama's position; he still holds the record for most immigrants deported in any presidental administration, 2.7 million: http://immigrationimpact.com/2017/01/04/deportation-numbers-2016/ [immigrationimpact.com]
(although Donald may be working on breaking that record)
Re: (Score:2)
Re: (Score:2)
That's either dumb of your or is overly partisan. The Obama administration deported more people than the Dubya administration. Stop getting your news from Infowars.
Dems vs Repubs [Re:So...] (Score:5, Insightful)
Actually, I've seen exactly the opposite analysis.
Democratic politicians want more legal immigration but less illegal immigration. Legal immigrants vote and pay taxes, illegals don't.
Republican politicians want less legal immigration, but more illegal immigration. Illegal immigration depresses wages, meaning more profits for corporations. (Even if the corporations don't hire illegals, the illegals have a downward pressure on all unskilled-labor wages).
Re: (Score:2)
Legal immigrants vote and pay taxes, illegals don't.
No, but the children that illegals give birth to do (and get automatic citizenship) vote, and I always hear Democrats and pro-illegal immigration tout how illegals 'pay taxes just like you and me' (SS, sales taxes, etc.). And they setup all of the sanctuary cities/states (you don't think those are for legal immigrants, do you?). To say that Democrats want less illegal immigration is just an ignorant statement.
Re: (Score:2)
Pence is far too much of a closet case homosexual
Being a happily married man faithful to his wife makes him a closet homosexual?
Let me guess, you invite other men in for a threesome then complain if your wife gets all the cock.
Re: Dems vs Repubs [Re:So...] (Score:2)
How is this in any way insightful?
1) it's CANADA, so really Republicans v Democrats is entirely irrelevant
2) your assertion about which party wants which immigration is entirely pulled from your ass. I could equally assert with as many or more examples of democrats defending and protecting illegal immigrants (Cf the last president) while Republicans are simply anti criminal.
Re: (Score:2)
How the above got modded "insightful" is absolutely beyond me. It is almost completely opposite of reality.
Democrats generally want more ILLEGAL aliens they can later TURN INTO voting residents who are dependent on "the system" through never-ending amnesty programs.
Republicans generally want more LEGAL aliens that will work for lower wages, yet pay taxes and not drain the system.
There aren't any amnesty programs [Re:Dems vs...] (Score:2)
Democrats generally want more ILLEGAL aliens they can later TURN INTO voting residents who are dependent on "the system" through never-ending amnesty programs.
That's silly. There aren't any "never ending" amnesty programs, there are no amnesty programs whatsoever that turn illegal aliens into voting residents,period. These don't exist.
The last time there was an amnesty program was under President Reagan, who provided amnesty for 3 million illegal immigrants. That was back in 1987, over thirty years ago. (that was the Immigration Reform and Control Act of 1986 [wikipedia.org], followed by Reagan's executive action to give legal status to more illegals [politifact.com] not covered by the Immigrat
Re: (Score:2)
Have you ever seen a member of MS-13 in real life? Do you have any idea, without googling it, how many members of MS-13 are in the United States? How sure are you that MS-13 isn't just another of the many mythical boogymen that authoritarians like Trump like to use to frighten people into acquiescing to authoritarianism?
[ed.note: there are fewer than 10,000 members of MS-13 in the United States, if you're willing to accept that MS-
Re: (Score:2)
Re: (Score:2)
The Long Island ones went to jail. There are a lot more Klansmen and neo-Nazis then MS-13 members, and they've caused a lot more violence.
Crime is crime. There was crime in Long Island long ago. I seem to recall some goombahs named "Gambino" that caused a lot of trouble.
Re: (Score:2)
While I'm sure there are some Canadians that think the US should accept all immigrants, it would be similar to the number of US citizens that think the US should accept all immigrants as well. Canada, or as a rule "Canadians" don't think any country should have a fully open immigration policy.
Re: (Score:2, Insightful)
Are you counting yourself too, or are you Native American?
"Native" Americans weren't native. They didn't pop up out of the ground in the Americas. They migrated here as well. Stop being a pedant.
Re: So... (Score:5, Funny)
Re: (Score:2)
Certainly many Mexicans were well established here in California long before the gringos showed up.
Re: (Score:2, Flamebait)
Certainly many Mexicans were well established here in California long before the gringos showed up.
You can tell who gets things done better when two countries are side by side yet have drastically different outcomes. For example Haiti and the Dominican Republic. Or the US and Mexico. Call us names all you want but there's no denying one society built much better things than the other.
Mexico would have been better off if the US annexed the whole thing. We would have a smaller border to defend and the whole of Mexico would have been more developed and prosperous. People forget that if you too many i
Re: (Score:2, Informative)
Mexico had alot of problems that were never replicated in the US. Spanish and English colonization was very different. [granburyisd.org]
Your remarks remind me of the times my Mother used to tell me that Israel turned desert into farmland. Remarkably ignorant of history.
Re: So... (Score:4, Insightful)
Must be nice to be ignorant of history. Mexico had alot of problems that were never replicated in the US. Spanish and English colonization was very different. [granburyisd.org] Your remarks remind me of the times my Mother used to tell me that Israel turned desert into farmland. Remarkably ignorant of history.
Everyone has their cross to bear but at some point they have to move on. I'm aware of history but I neither use it as an excuse nor an accusation. All countries have behaved badly in some ways at some times, are we all victims or do we move on and try and make things better?
Irrelevant (Score:5, Insightful)
It doesn't matter whether the "RND" function is ideally random in a mathematical sense. It only matters whether the "random" number generated is independent of the identities of the people applying to be admitted.
Re:Irrelevant (Score:5, Insightful)
Exactly. And if the order in which the data is inserted into Excel is essentially random, that's good enough. There is no fair or unfair if the number of applicants exceeds the quota and there is no opportunity to game the system (such as naming yourself Aaron A Aaronson to appear first, or something)
Re:Irrelevant (Score:5, Funny)
such as naming yourself Aaron A Aaronson to appear first, or something
[Tears up application]
Re:Irrelevant (Score:4, Insightful)
The bigger question to me is why are they using Excel for this? Spreadsheets are for calculating things. They are absolutely the wrong tool if you need the data you're working on to remain consistent or auditable. An immigration employee could take the spreadsheet after the random numbers were assigned, and copy-paste names or random numbers around to move people to/from the selected and denied categories, and there'd be no way to detect they'd done this.
Re: (Score:2)
In a perfect world, perhaps we'd worry about something like this.
You can always count on the government to be too stupid for words, until there's profit involved.
Immigration Canada:
Re:Irrelevant (Score:5, Insightful)
While that's true, if Excel's RNG results in a pattern (e.g. cell A2534 is always assigned a low number and thus selected),
If you had bothered to read the finr article, you'd have learned that the tempest in the teapot is that Excel uses a pseudorandom generator. That's just like what a lot of systems use. It costs time and money to do real random generation, and it requires some hardware. PGP, IIRC, requires someone to type at the keyboard and it times the characters to generate a random number. There are radiative decay RNGs. There's even a lava lamp based RNG.
But MOST of the "random" for most software is pseudo. And the algorithms are published.
The fine article talks about how bad some RNG is because you could "reverse engineer" the algorithm. In 1980 I tested the RNG in DEC RSX-11, and I simply looked up the algorithm in the manual to see if the results from testing matched theory. In a manual. "This is how the RAND function works..." As I recall, it was based on doing a simple calculation on a double precision number and pulling the middle 32 bits out of the number as the random output. You could actually write the code to do this yourself.
it could result in immigration employees who know of this gaming the system, to do an immigrant friend a favor or even auctioning the spot to the highest bidder.
Oh for pete's sake. If an immigration employee has this much access to determining who wins and who loses, then even if the RNG is a true, completely unpredictable, physical random process based number, all he'd have to do is run the random generation process over again until his chosen winner "won".
The bigger question to me is why are they using Excel for this? Spreadsheets are for calculating things.
Yeah, calculating things. Like random numbers. They're using Excel because it works for this and didn't cost them $1 million to pay a consultant to write something in python to do the same thing.
An immigration employee could take the spreadsheet after the random numbers were assigned, and copy-paste names or random numbers around to move people to/from the selected and denied categories,
They could do this even if the RNG is truly random, so the RNG has nothing to do with the problem.
Re: (Score:3, Insightful)
They're using Excel because it works for this and didn't cost them $1 million to pay a consultant to write something in python to do the same thing.
A million dollars?
$ od -d /dev/urandom | less
I'm in the wrong job.
Re: (Score:2)
The whole concept is a stupid as fuck. Have a large list of people, who a very similar (we are not all the same), in measured characteristics, introduce more characteristics until sufficient difference is found to be more selective. In this case, simply get them to participate in an IQ test, the winners win. Quite reasonable working upon the basis that all other measured characteristics are equal. Only US law enforcers think they can win by going for lower intelligence (civil court costs should have well an
Re: (Score:2)
IQ testing isn't terribly useful. For instance a highly intelligent person that hasn't had access to education will score worse than someone like yourself that has.
Re: (Score:2)
Implementing a proper RNG isn't expensive or time consuming at all. Windows has one built in, that continually gathers entropy from things like user input, system timers, I/O timing and the like. Additionally, most modern CPUs have a built in hardware RNG. In fact, even the CPU on a Raspberry Pi one.
The issue is that Excel uses a crappy old PRNG that is known to be quite bad. Presumably this is done for compatibility reasons. This RNG does not produce an even spread of numbers over the entire range, and wit
Re: (Score:2)
If you don't trust the person behind the spreadsheet, a better RNG isn't going to help.
They could run a perfect RNG, and then just move the person of their choice to the top of the list.
If you want to avoid fraud, you need to compartmentalize the process.
Re: (Score:2)
Itâ(TM)s going to depend on both the number of rows (unknowable in advance) and probably the current time. At least that was the common seed to use, historically.
Anyway, if someone suitably positioned wants to sell access, they can just skip the whole random spreadsheet shenanigans and sell the person a permanent resident card directly.
Re: (Score:2)
Yes, this is it. If the order in which people are assigned their "random" numbers is random, then the issue is moot.
Re:Irrelevant (Score:5, Insightful)
That's kind of my thoughts on it, too. And who is at a disadvantage in this scheme? Is it the person entered in row 13,428? Or anybody who was entered on July 16th? The chances of ending up at an advantage or a disadvantage are themselves probably pretty random.
Re: (Score:2)
The advantage/disadvantage depends on the range of the random number generated. Since it's a random number, duplicates can be created. As an example, applicants who end up in the 9,997th through 10,004th positions all have the same random number. How those eight records are sorted makes a huge difference in outcome.
I just did a quick random number assignment in Excel for 20 records. First thing I noticed is that if the random function is used in the cell, sorting the data caused new random numbers to be gen
even weaker (Score:5, Insightful)
The story is about an issue that is completely irrelevant.
It doesn't matter whether the "RND" function is ideally random in a mathematical sense. It only matters whether the "random" number generated is independent of the identities of the people applying to be admitted.
It isn't even that. Just because the distribution of random numbers isn't random it doesn't mean the sort order based on that isn't random. For example, suppose my random number generator only put out numbers divisible by 1/(2^16) which is what a finite precision binary based system is going to do. This distribution isn't random because it's zero density at many possible floating point values. Yet the sort order might be perfectly random.
Re: (Score:2)
Or, what if the random number generator only picked numbers divisible by three? Or always alternated an odd number with an even number? That's not random at all-- but it doesn't affect who gets picked to immigrate.
Re: (Score:2)
The story is about an issue that is completely irrelevant.
It doesn't matter whether the "RND" function is ideally random in a mathematical sense. It only matters whether the "random" number generated is independent of the identities of the people applying to be admitted.
It isn't even that. Just because the distribution of random numbers isn't random it doesn't mean the sort order based on that isn't random. For example, suppose my random number generator only put out numbers divisible by 1/(2^16) which is what a finite precision binary based system is going to do. This distribution isn't random because it's zero density at many possible floating point values. Yet the sort order might be perfectly random.
Even further, it appears that the flaw with Excel's PRNG is that, if you know the seed, then all of the resulting numbers are predictable. So, for example, if you can duplicate the starting seed, then it's trivial to generate the exact same list of 100,000 random numbers for their sort function. According to a linked article within TFA:
"The generator is run in user mode rather than in kernel mode, and therefore it is easy to access its state even without administrator privileges. The initial values of part of the state of the generator are not set explicitly, but rather are defined by whatever values are present on the stack when the generator is called.
The cryptographers discovered that the state of the generator is refreshed with system generated entropy only after generating 128KB of output for the process running it. "The result of combining this observation with our attack is that learning a single state may reveal 128KB of the past and future output of the generator," they explain.
But that's irrelevant to this process. They're not using the same numbers generated previously. They're not providing their starting values at the beginning of the year so th
Re: (Score:2)
it appears that the flaw with Excel's PRNG is that, if you know the seed, then all of the resulting numbers are predictable.
That's how all PRNGs work.
Re: (Score:2)
"independent of the identities"
Why do I suspect that is the real problem here?
Re: (Score:2)
The story is about an issue that is completely irrelevant.
It doesn't matter whether the "RND" function is ideally random in a mathematical sense. It only matters whether the "random" number generated is independent of the identities of the people applying to be admitted.
Also, how random does something need to be before it is acceptable and for what application? For many applications being sufficiently random is enough. Simply showing that it is not a predetermined sequence that can't be easily gamed is usually acceptable.
Anti-Luck Discrimination (Score:2)
It only matters whether the "random" number generated is independent of the identities of the people applying to be admitted.
Indeed, if it was a true random number it would end up discriminating against unlucky people. Something the poor sod having to use an Excel spreadsheet of this magnitude probably understands all too well now.
Re: (Score:2)
"Unbiased" is what you're going for.
Even so, this is the sort of detail that a system designer ought to get right, even if he thinks that there probably won't be troublesome consequences if he gets it a little bit wrong.
When in doubt, a quality algorithm is always better than a bad one. This is the thing about security; when the bad guys figure out an angle to your sloppiness you've overlooked, you lose.
Re: (Score:2)
It's not a random number. Not even close. It's very much predictable.
For this particilar application I can't really see how it could be exploited, as I assume Excel does a decent job of seeding the generator.
The question really is:
Why is there a class/method/function/library named Random when it's not random? It's the same for Java,
Re: (Score:2)
The story is about an issue that is completely irrelevant.
Not completely irrelevant. It indicates that some government cogs are using Windows, machines that will soon be owned by Chinese and/or Russian government cybercrooks, if not already.
Fake News (Score:4, Insightful)
The story is about an issue that is completely irrelevant.
It doesn't matter whether the "RND" function is ideally random in a mathematical sense. It only matters whether the "random" number generated is independent of the identities of the people applying to be admitted.
With no intention of diminishing the importance of your statement; that is blindingly obvious. There are two other excellent points raised by others in the comments here: that imperfect randomness does not make the process manipulable by immigration candidates and that sort order of assigned imperfect random numbers can itself be perfectly random.
The story is mis-reported as a scandal; there is in fact no scandal whatsoever. So who made up the fake news? Tom Cordoso is the author of the original story [theglobeandmail.com] at the Globe and Mail which the Gizmodo article linked in the Slashdot summary cites. Cordoso quotes Université de Montréal computer-science professor Pierre L’Ecuyer as saying “Anything would be better” [Than the Excel random number generator] but, crucially, Cordoso omits the context of that comment. Was L’Ecuyer referring to its suitability for this particular method and application, or was he commenting on its suitability for general use, including, for example cryptography? In neither the Gizmodo nor Globe and Mail articles can I find any mention of an expert unambiguously expressing judgment on the immigration randomization method specifically. A close reading suggests that the criticism originates with the journalist, and that he deceptively implies it to be the opinion of experts.
Some enterprising citizen journalist should contact the cited experts and ask them 1) Did their comments refer to general usage of the Excel random number generator or specifically to the immigration randomization methodology. 2) What is their opinion of the immigration randomization methodology 3) Do they agree with the points made here about it being a nothingburger 4) Have they read the Globe and Mail article, if so do they believe that their comments were wrongly contextualized.
If anyone does that, it would be nice to see a followup article here on Slashdot.
We have a case for... (Score:2)
Re: (Score:2)
RAND isn't "perfectly" random, what about the list (Score:2)
Well I've ready before about how the RAND() function of Excel 2003 and 2007 wasn't good enough for scientist purpose. But I seriously wonder what's the bias and how it'll affect a 10 000 number scale. I don't think it's candidate #1455 have 5 time more chance to be picked than candidate #976. I guess it's more in the "0.0001 time more" scale.
Furthermore, how is the list order selected? Because if the order of the list is "kinda" random, it add the the randomness of the process. In other word, if the list or
Re: (Score:2)
"Randomly" sort the list 10 times and everything will end up "randomer".
Re: (Score:3)
Where's the issue? (Score:2)
Unless I'm missing something, it doesn't matter how random the PRNG is if the selection isn't influenced by the other relevant data. Everyone got their random number generated by the same shitty PRNG, so it's a fair and equitable system.
sounds like good news to me (Score:2)
not the issue (Score:3)
As long as no one knows what the biases are, there is not an actual issue. Probability, at least for these purposes, is epistemological.
That said, they should not use proprietary software. Public money, verifiability, freedom and so on.
100,000 applicants since when? (Score:2)
What did they do back when Excel could handle only ~65K rows?
Re: (Score:2)
Re: (Score:2)
They used columns. It's a spreadsheet, not a spreadline.
Is that some kind of attempt to be funny? Epic fail since the column limit was 256 back when the row limit was 65K.
Random enough (Score:5, Interesting)
Regardless of Excel's poor random function, the way this is being described as being done, it sounds pretty legit and random enough. There's no bias on assigning the random number to each name, and the name itself isn't being used to generate the random number. So this should be fine.
Just because it doesn't meet some math/computer geek's standards of proper random number generation, doesn't mean it's not useless for this application. I say thumbs up. The RNG being perfect isn't really necessary.
So what? (Score:2)
"Random" in this application means the numbers assigned to each applicant are generated by the RAND function, then chosen sequentially from the resulting list.
If instead every applicant was assigned a sequential number, then the RAND function was used to pick from that list, then it is possible that certain sequential numbers would have a less equal chance of being selected, but not under the reverse.
If the RAND function assigns multiple users the same 'random' number, so what? All duplicates get selected a
Who says it's not random? (Score:2)
Sure, RAND() might only be pseudorandom but they're putting the numbers in the spreadsheet in a random order. It doesn't sound like they're sorting the application numbers before assigning them a pseudorandom number.
Just do the sort... (Score:2)
What random actually means... (Score:2)
Nominally, random numbers are generated through a true
Re: (Score:2)
Well, if the applicants are initially entered based on the date their application was received, then clearly certain days of the year could be "golden", and certain ones "bad"...so yes, this could become exploitable....
Re: (Score:2)
This story is muckraking bunk by people who again don't really want people to understand security as much as they want to stamp a name for themselves. I'd be much more concerned that this is being handled in a spreadsheet rather than in an air-gapped database infrastructure.
Personally I am not even concerned about them using a spreadsheet, actually it is a very good way to implement it in a cost effective matter given the type of data we are talking about here. The whole article is complete garbage as unless you can determine exact time a spreadsheet is going to be run you can't really manipulate the results and if you could manipulate the results at that point it doesn't matter whether the results are random or not as a true RNG is not going to stop someone manipulating the r
Re: (Score:2)
This story is probably pushed by Big Random, who wants to get contracted to write a better system.
List is already random (Score:5, Insightful)
Unless they let entrants pick where they are in the list, it doesn't matter if the random number generator is not completely fair.
Maybe it's biased such that entrants 50,000 - 51,000 are much more likely to end up sorted to the top, but unless the entrants can choose where they are in the list, I don't see why that really matters. Sure, someone that controls the list could move their friends to that range to make them more likely to end up at the top, but they could also move their friends to whatever random numbers and up at the top.
Translation: (Score:5, Funny)
Gizmodo just discovered what a PRNG is
Old system (Score:3)
What's wrong with first come, first serve?
Re: (Score:2)
And more importantly, why is a random pick more fair than that?
Re: (Score:2)
Exactly. This Excel crap is certainly random enough for randomness. The real question is whether an immigration system SHOULD be random. Is that in the best interest of the country? Of the citizens? Of the economy? What is "fair" and to whom? Does a country "owe" fairness or randomness to those seeking entry? Should those with better skills, education, or knowing the local language be prioritized? Should those with existing family support structures in the intended country be prioritized? These a
Still random?!! (Score:2)
Re: Still random?!! (Score:2)
Unlucky need not apply (Score:2)
So, Canada just does not want unlucky immigrants...
That said, I'm shocked, shocked to find out, the most adorable country in the world accepts only about 10% of the immigrants seeking to enter (legally)...
Re: (Score:2)
That said, I'm shocked, shocked to find out, the most adorable country in the world accepts only about 10% of the immigrants seeking to enter (legally)...
This is for P.R. Status given for family reunification purposes. Permanent Resident status is roughly equivalent to the US Green Card.
Real random number generator (Score:2)
Why a Wall Full of Lava Lamps Is a Terrific Random Number Generator [popularmechanics.com]
PRNG algorithm depends on the Excel version (Score:2)
The article and embedded links talk about how bad Excel's algorithm is, but never states the Excel version that Canada's IRCC uses. In this case it matters because recent versions of Excel are OK.
Excel 2010 and later uses Mersenne Twister for the PRNG. This is good.
https://support.office.com/en-... [office.com]
Excel versions before Excel 2010 use an implementation of the Wichman-Hill that provides not-so-good pseudorandom numbers.
https://support.microsoft.com/... [microsoft.com]
Re:Trump is a traitor (Score:4, Funny)
"When you are dead, you do not know you are dead. It's only painful & difficult for others. The same applies when you are stupid." --someone
Re: (Score:3)
Some people will tell you that quantum uncertanties introduce randomness, but my theory discounts that.
Please tell us more about your theory. I'd be interested in reading about it. And if I sound snarky, sorry, but I was predestined by the universe to ask you that question.
Re: (Score:3)
The initial conditions for the Universe were set at the Big Bang, and everything that follows is deterministic.
This demonstrates a rather painful lack of understanding of quantum mechanics, which generally accepts that God does play dice (or as Niels Bohr put it, stop telling God what to do.)