Have Over 1 Million URLs & Need to Remove Duplicate URLs & Domains? Here You Go!
2014-01-24, 01:13 PM,
#1
This thread appears menacing & long. While it is long, it's only this long because I have a tendency to be way overly detailed. What you see below is INCREDIBLY easy and has very few steps. I just wanted to make sure everybody understood.

I scrape over 80 to 90 million URLs a week using ScrapeBox & other tools. While that may sound great, until I had a working solution to what the title of this thread is, it was a nightmare.

Having to import 1 million URLs at a time into ScrapeBox, then removing duplicates, saving the list & doing that over 80 times, then combining those new files, splitting them into chunks of 1 million & then doing it 60 times more...repeat, repeat, repeat...until I finally got a list that had no duplicates...well, it seriously takes HOURS & HOURS and anybody working with large lists knows that.

Well, here is how you can take ALL that headache away & simply take 10 million URLs (or more...or less) & using ONE program, remove all duplicate URLs & duplicate domains in less than 10 minutes. Sounds good right?

First, you'll need a couple things (they are free).

Text Collector: Download Page If you're using ScrapeBox & you've harvested more than 1 million URLs, you've likely got TONS of different .txt files all with X amount of URLs in them.

You're going to want to get ALL of those URLs into one .txt file. That software I've just linked to will do that for you.

gVim: Download Page gVim is GREAT for working with ENORMOUS .txt files. It can handle opening files with over 10 million lines no problem...it's also what is going to make removing duplicate URLs & domains a breeze.

Once you've got those two things, you're ready to get started.

Go ahead & combine all your .txt files that contain all your urls into one file.

Right click on that new (huge) .txt file & click on "Edit with Vim"

Once gVim has opened, you'll know it is "ready to go" (and you can get started) if you see a flashing cursor in the uppermost left hand corner. If you don't, wait until you do (it can take some time depending on your computer & it's processing power).

Here is a picture of the interface you'll see. In the green circle in at the bottom, that's the number of lines in the file.
[Image: gviminterface.jpg]



REMOVING DUPLICATE URLS


Now, to remove duplicate URLs, simply type the following (you don't have to click anywhere, just start typing it)

Register or login to see this content

As soon as you type the : key, you'll see the cursor jump to the bottom left hand corner of the screen. Keep typing the rest, exactly as I showed you above in red.

Once you've typed it out, press enter. It will proceed to remove all duplicate URLS & sort them alphanumerically. This can take awhile depending on the number of URLs.

If you'd like to see how it functions quickly, just open up a small file with a couple hundred URLs.

It's really that simple. Once it's complete, you'll see an updated number of lines at the bottom of gVim's interface & it will tell you how many lines it removed.
Latest Thanks - View all

bartolhc(2017-04-01 03:16 PM) Radiit(2016-09-21 01:40 PM) ngonampc(2016-08-15 09:49 AM) johnsaka(2016-08-01 06:32 PM) tonnyba111(2016-07-21 11:20 AM) gigstick(2016-03-04 06:48 PM) imammeheditk(2016-02-29 05:15 AM) dtk240588(2015-06-15 01:47 PM) xenofongr(2015-06-11 12:27 PM) jacksparow119(2015-03-21 10:23 AM) WalkingDead(2015-03-13 05:28 AM) Fierce(2015-02-11 09:07 AM) TriVixXx(2015-02-07 10:12 PM) Vadovas(2015-02-04 08:46 PM) Dr.Alekhine79(2015-02-03 12:20 PM) Christer(2014-12-20 10:59 PM) zupodes(2014-11-29 03:12 PM) kristylove(2014-11-10 11:31 PM) netbull(2014-11-06 11:40 AM) spAcebound(2014-11-06 05:35 AM) alexjer12(2014-10-29 09:06 PM) abef6600(2014-10-14 01:29 AM) Dobud(2014-10-11 07:43 PM) selfstarter1(2014-10-07 11:16 AM) bmn(2014-09-30 09:50 AM) rubelpangsa(2014-09-29 01:06 PM) coletrain(2014-09-14 11:27 PM) pepito(2014-09-08 07:07 PM) delmo(2014-08-29 09:29 AM) krystianbac(2014-08-28 03:09 PM) rasturit(2014-08-22 11:08 AM) Nevill(2014-08-22 03:23 AM) roliviu(2014-08-20 08:32 PM) johneyboy(2014-08-16 11:16 AM) skynet32(2014-08-15 04:18 AM) palseif(2014-08-14 03:50 PM) guuck(2014-08-14 12:14 AM) molabee(2014-08-12 03:11 PM) limang(2014-08-08 10:16 PM) driftpower(2014-08-08 09:12 PM) mgmc(2014-08-08 03:25 PM) admindre(2014-08-06 01:38 AM) BeHappy(2014-08-05 07:01 PM) digall(2014-07-24 12:43 PM) prigma(2014-07-16 01:52 PM) hollycraze(2014-07-15 04:59 PM) besupreme(2014-07-09 12:06 AM) bbc554(2014-07-03 04:57 PM) Geralt(2014-07-01 02:35 PM) panzerbravo(2014-06-30 03:06 AM) Super(2014-06-24 06:05 AM) zakir375(2014-06-20 10:32 PM) jim888(2014-06-20 10:56 AM) mrkenny(2014-06-17 03:12 PM) yazisback(2014-06-12 12:29 AM) miteshinfocus(2014-06-09 05:16 PM) JejakaNoName(2014-06-08 08:53 PM) kimple89(2014-06-08 03:28 PM) HurAbbas(2014-06-07 10:09 PM) ilkseo(2014-06-07 08:16 AM) hackskeygen(2014-06-02 05:56 PM) darkdream(2014-05-31 04:19 AM) Mobeen123(2014-05-28 12:20 AM) hopsa(2014-05-27 03:05 AM) teers44(2014-05-25 08:54 PM) abaddon013(2014-05-18 10:09 PM) mantuxas(2014-05-17 08:00 AM) liyang325500(2014-05-15 04:55 PM) Silverdot(2014-05-12 03:14 PM) Vampirion_LKM(2014-05-12 04:09 AM) manusiaonline(2014-05-11 07:07 PM) Taajir(2014-05-11 02:57 AM) stevewonderburg(2014-05-07 08:56 PM) blackseocn(2014-05-07 08:09 PM) javablankon(2014-05-07 07:13 PM) djaskol(2014-05-06 11:42 PM) alexusua(2014-05-06 09:35 PM) SamoX(2014-05-05 03:28 AM) Pawepeks(2014-05-05 12:59 AM) mikederoo(2014-05-04 06:38 PM) sakr55(2014-05-04 03:40 PM) kungfu.824(2014-05-04 03:01 PM) Trai(2014-05-04 02:20 PM) deimprover(2014-05-04 06:48 AM) kicku(2014-04-29 09:59 PM) LoveHQ(2014-04-28 11:56 PM) jonykatro(2014-04-28 02:33 PM) Leunamriu(2014-04-12 11:15 PM) wqaxsz(2014-04-12 03:29 PM) kakami(2014-04-12 03:17 PM) dennisgrazzini(2014-04-12 07:38 AM) noob1(2014-04-11 02:44 PM) pwned(2014-04-11 12:31 AM) lucky2002(2014-04-10 06:59 PM) leo7068(2014-04-09 12:54 AM) hianaxes(2014-04-08 09:41 PM) Moneypoolm(2014-04-08 01:40 AM) aniket93(2014-04-08 12:26 AM) staffer(2014-04-07 11:27 PM) WaX(2014-04-07 03:46 PM) InboXer(2014-04-07 04:27 AM) NewUser(2014-04-07 04:02 AM) mnhweb(2014-04-06 01:29 PM) abdoahmed(2014-04-06 01:26 AM) longluxi(2014-04-05 09:20 AM) CatD3vjl(2014-04-02 10:15 PM) Motivation(2014-03-31 08:01 PM) soad(2014-03-31 01:53 AM) Conley18(2014-03-30 09:52 PM) Red John(2014-03-29 03:25 AM) ant1kan(2014-03-29 12:15 AM) pratham7(2014-03-28 03:46 PM) hitcomvn(2014-03-28 11:33 AM) RodrigoBorgia(2014-03-27 10:55 PM) Alexich(2014-03-26 10:51 PM) solobeton(2014-03-24 03:29 PM) ifreaks(2014-03-23 11:47 PM) Tuw(2014-03-23 02:09 AM) Rodimus(2014-03-23 01:31 AM) L0ve(2014-03-21 05:39 AM) krzysiek151(2014-03-19 08:49 PM) Incomeme(2014-03-19 02:06 PM) domin(2014-03-19 06:38 AM) Sunspear(2014-03-19 02:23 AM) shiaboxs2121(2014-03-18 01:48 AM) Florentin(2014-03-16 01:12 PM) MikeMac(2014-03-16 12:58 PM) CEED(2014-03-16 11:35 AM) lvcash(2014-03-16 12:07 AM) kapitan(2014-03-15 09:42 PM) Jaguar(2014-03-15 02:12 PM) Spectre(2014-03-15 12:07 PM) kancan(2014-03-15 11:55 AM) Grizzly(2014-03-12 06:17 PM) nexusnetcafe(2014-03-11 04:12 AM) mpaz(2014-03-10 07:52 AM) crashactiv(2014-03-10 04:29 AM) wrcja(2014-03-07 08:54 PM) seomister(2014-03-07 01:40 PM) Thelegend(2014-03-04 12:56 AM) wooot(2014-03-03 12:33 PM) TheSilentKid(2014-03-01 10:26 PM) BadSpawn(2014-03-01 06:50 AM) blkmagic(2014-02-28 11:34 AM) mrAS7(2014-02-27 11:16 PM) Ang3l1n(2014-02-27 05:17 PM) Dragoshhh(2014-02-26 02:30 AM) Windmm(2014-02-24 05:22 PM) DeusEx(2014-02-24 03:45 PM) icyteru(2014-02-24 03:32 PM) lcascisf(2014-02-24 02:14 PM) shaderviner(2014-02-23 05:46 AM) albahja(2014-02-23 03:10 AM) sloyen93(2014-02-22 10:03 PM) Rift(2014-02-22 11:03 AM) darkvivi(2014-02-21 10:18 PM) TheMagic(2014-02-21 05:16 AM) N1mrod(2014-02-21 03:31 AM) asgardian(2014-02-21 02:19 AM) thecpaprincess(2014-02-21 01:13 AM) Panamera(2014-02-20 11:37 AM) peter121(2014-02-20 09:14 AM) ganars(2014-02-20 09:10 AM) ScottyAdamz(2014-02-20 08:03 AM) TheGreeneXXX(2014-02-20 01:10 AM) dimafatality(2014-02-20 12:39 AM) fomalstar(2014-02-19 10:01 PM) sina26(2014-02-19 06:14 AM) SoFt(2014-02-19 03:22 AM) trebol(2014-02-19 01:25 AM) alewy(2014-02-18 06:55 PM) unknownman(2014-02-18 09:22 AM) sam1983(2014-02-17 10:36 PM) gokhan(2014-02-17 08:19 AM) dai3504(2014-02-17 07:31 AM) giobeats(2014-02-17 05:06 AM) spAceflight(2014-02-17 03:06 AM) imdavidluu(2014-02-16 12:47 PM) -qwerty-(2014-02-13 11:35 AM) krypton(2014-02-13 08:52 AM) Number23(2014-02-13 08:47 AM) zilvis89(2014-02-13 02:28 AM) anthonytn(2014-02-11 09:55 PM) skekeu(2014-02-11 04:15 AM) racker12(2014-02-10 07:19 PM) witcherek(2014-02-10 06:53 AM) mcsquare(2014-02-10 02:03 AM) kimiwizz(2014-02-09 05:40 PM) Bogarus(2014-02-09 03:42 PM) jayesh1486(2014-02-09 01:11 PM) xitoxic(2014-02-09 01:26 AM) kinlee(2014-02-08 03:04 AM) N/A(2014-02-07 01:23 AM) assenal(2014-02-06 10:23 PM) Marian1995(2014-02-06 03:01 PM) Daniel Robyn(2014-02-06 02:38 PM) dreaded66(2014-02-06 01:29 PM) MrWest(2014-02-06 12:08 PM) Mesina(2014-02-06 08:57 AM) djkrmak02(2014-02-06 06:39 AM) SaRNEh(2014-02-05 09:30 PM) Aamobu(2014-02-05 06:40 PM) xploader(2014-02-05 04:20 AM) Delfo(2014-02-05 04:02 AM) stevenvertigo(2014-02-04 05:44 PM) cracky(2014-02-04 11:42 AM) Tschitschikow(2014-02-04 11:39 AM) yonswac(2014-02-04 08:55 AM) RisingShell(2014-02-04 07:00 AM) gdfgdf(2014-02-04 02:19 AM) rutarutule(2014-02-03 07:49 PM) yuba1977(2014-02-03 06:21 PM) dupdomain(2014-02-03 03:28 PM) Ripko(2014-02-03 07:17 AM) jsxz(2014-02-02 10:11 PM) KoTaPaKa(2014-02-02 04:09 PM) typhone(2014-02-02 05:10 AM) makosgsi(2014-02-01 07:02 PM) cristianhell(2014-02-01 04:11 PM) Joker(2014-01-31 11:39 AM) firdausmc(2014-01-30 10:16 PM) ihasdevkit(2014-01-30 11:12 AM) guanlo(2014-01-30 10:16 AM) Daivanas(2014-01-30 04:32 AM) paulzai82(2014-01-30 04:06 AM) Aidan(2014-01-30 12:49 AM) 15challenger(2014-01-29 09:40 PM) konker5(2014-01-29 08:44 PM) DondotheGreat(2014-01-29 08:34 PM) JustNick(2014-01-29 03:08 AM) paulns(2014-01-28 11:41 PM) Viper(2014-01-28 12:17 PM) WalterWolf(2014-01-27 07:44 PM) dinkolukic(2014-01-27 07:06 PM) KilluminatiX(2014-01-27 04:21 PM) crespozooo(2014-01-26 04:16 PM) »Wave«(2014-01-26 03:26 PM) sidaz10(2014-01-26 03:00 PM) kinaks(2014-01-26 09:59 AM) capitalistpig50(2014-01-26 09:50 AM) J__128(2014-01-26 09:17 AM) King Of Kings(2014-01-26 05:24 AM) OvidiuR94(2014-01-26 04:54 AM) Һ૪८૯Ր(2014-01-26 02:18 AM) luckystomp(2014-01-26 01:49 AM) ashu912(2014-01-25 11:55 PM) smeak(2014-01-25 11:23 PM) 08521081(2014-01-25 09:51 PM) ambros23(2014-01-25 08:50 PM) daph(2014-01-25 05:12 PM) vahnz13(2014-01-25 12:15 PM) ιηvιηcιвℓε(2014-01-25 11:29 AM) tbpi007(2014-01-25 10:53 AM) Mastino(2014-01-25 05:01 AM) tomyslav(2014-01-25 04:26 AM) necvader(2014-01-25 02:01 AM) markethealth02(2014-01-25 01:09 AM) mean969(2014-01-25 12:41 AM) vendyz(2014-01-24 07:45 PM) Thunderstorms(2014-01-24 07:31 PM) ImAlanas(2014-01-24 07:14 PM) Yaveril(2014-01-24 05:20 PM) Josh C.(2014-01-24 04:43 PM) janonymous(2014-01-24 04:42 PM) forthewin123(2014-01-24 04:08 PM) ★★★ BotMan ★★★(2014-01-24 03:52 PM) Ernegie(2014-01-24 03:26 PM) Shly(2014-01-24 03:20 PM) unu(2014-01-24 03:16 PM) Mind the Gap(2014-01-24 03:14 PM) CapsLuck(2014-01-24 02:56 PM) rajuonline(2014-01-24 02:49 PM) KCMM(2014-01-24 02:23 PM) J a Y(2014-01-24 01:48 PM) ichief(2014-01-24 01:43 PM) jhakasseo(2014-01-24 01:28 PM) aceyx3(2014-01-24 01:23 PM) Ashley(2014-01-24 01:20 PM) f0rest(2014-01-24 01:15 PM) 
Reply
2014-01-24, 02:50 PM,
#2
nice tutorial dude. I am going to try this with your list.
Latest Thanks - View all

manusiaonline(2014-05-11 07:08 PM) shiaboxs2121(2014-03-18 01:49 AM) ddesk52(2014-03-09 10:47 PM) xploader(2014-02-05 04:18 AM) 
Reply
2014-01-26, 01:50 AM,
#3
Thank you,
had some issues with texts too.
Reply
2014-01-26, 01:56 AM,
#4
I've never tried it with that many URL's, but there is an extension for Notepad++ called TextFX in which will remove duplicates. I've only tried it with a 2,000,000 URL list, so it could be different. O.o

[SHARE] CPA Elites Forum Citer Script! Look Inside! [Forum Tool]
[SHARE] CPA Elites Link To Post PM Script! Look Inside! [Forum Tool]
[Image: D8HZ2ho.png]
"I have not failed 1,000 times. I have successfully discovered 1,000 ways to NOT make a light bulb." -Thomas Edison
Reply
2014-01-26, 02:12 PM,
#5
(2014-01-24, 02:50 PM)rajuonline Wrote: nice tutorial dude. I am going to try this with your list.

:D

(2014-01-26, 01:50 AM)luckystomp Wrote: Thank you,
had some issues with texts too.

What kind of issue you got
please post below

(2014-01-26, 01:56 AM)OMGWTFISTHIS Wrote: I've never tried it with that many URL's, but there is an extension for Notepad++ called TextFX in which will remove duplicates. I've only tried it with a 2,000,000 URL list, so it could be different. O.o

well this one works pretty fast never tried notepad++ extension
but this one helps me to remove duplicates from 5,400,000 URL List within 10 minutes
Reply
2014-01-26, 03:33 PM,
#6
(2014-01-26, 01:56 AM)OMGWTFISTHIS Wrote: I've never tried it with that many URL's, but there is an extension for Notepad++ called TextFX in which will remove duplicates. I've only tried it with a 2,000,000 URL list, so it could be different. O.o

i ald installed the plugin
how to remove the duplicate? tq
Reply
2014-01-27, 08:58 AM,
#7
nice tutorial my friend. goinna try this with your list.
Latest Thanks - View all

Jain(2014-01-29 08:32 PM) 
Reply
2014-01-29, 08:32 PM,
#8
(2014-01-27, 08:58 AM)MarlonKS Wrote: nice tutorial my friend. goinna try this with your list.

sure
it really works
Reply
2014-02-06, 07:15 AM,
#9
i got red problem that says out of memory :(

Looking for way to rank on youtube
Reply
2014-02-07, 10:14 AM,
#10
(2014-02-06, 07:15 AM)djkrmak02 Wrote: i got red problem that says out of memory :(

if it is in gsa lower the numbers of threads
this problem is not related to the list it's with your PC speed
Reply


Possibly Related Threads...
Thread Author Replies Views Last Post
  Data URLs - hide HTML inside a link (and other goodies) styx 17 1,857 2017-05-06, 04:22 PM
Last Post: styx
  Buying expired domains with backlinks. nudes 10 2,450 2016-08-25, 11:29 AM
Last Post: 1linklist
  1,7 mln urls from 2016 to seo jeff1995 10 2,468 2016-06-04, 01:06 AM
Last Post: jeff1995
  Do you think using AMP will affect rankings in SEO? The_Mentor 2 951 2016-03-03, 06:51 AM
Last Post: fnddiego02
  We Analyzed 1 Million Google Search Results. Here’s What We Learned About SEO HawkEye 9 1,271 2016-02-29, 09:26 PM
Last Post: marketingstar





About Us | Contact Us | CPA Elites | Advertise | Stats | Staff Team

© 2013-2017 CPA Elites Ltd
Enhanced by MyBB and WallBB
Return to top