Orange has almost no perfect rhymes. The only word in the 20-volume historical Oxford English Dictionary that rhymes with orange is sporange, a very rare alternative form of sporangium (a botanical term for a part of a fern or similar plant). Silver is another word for which it is almost impossible to find a perfect rhyme: the only candidate is the rare word chilver, which the 20-volume Oxford English Dictionary defines as 'a ewe-lamb' (i.e. a female lamb). Both orange and silver do have half-rhymes, though: the Oxford Rhyming Dictionary gives lozenge as a half-rhyme for orange, for example, and salver as a half-rhyme for silver.
What's the difference between a full rhyme and a half-rhyme? A full and stressed rhyme (e.g. hand / stand) or even an unstressed rhyme (such as handing / standing) contain vowels that are common to both words, while a half-rhyme like orange / lozenge or silver / salver has obvious differences between the vowels in certain syllables. The technical term for a half-rhyme is 'pararhyme'.