Get all the information you need to find and apply for these legitimate work at home jobs, and get all the resources you need for becoming a successful rater
If you are looking for a list of projects out there from companies like Lionbridge and Appen, we have them. As you may know Appen acquired Leapforce in 2017, since then the company has decided to close it is own project management platform known as Appen Global and move all projects to Appen connect. The group of projects known as Falcon (mainly Facebook-related projects) now is no longer actual, these projects also have been moved to Appen connect platform.
Taking into account the fact that all projects have code names and little description is provided, many candidates might be lost in terms what project to go for.
Bellow you will find a list of most popular projects and links to more info about each project:
Provide relevance and accuracy ratings for one of the world's largest and most popular mapping applications.
The job has an ABOVE average volume of work.
Maps Evaluation Project
Provide relevance and accuracy ratings for one of the world's largest and most popular mapping applications.
The job has an ABOVE average volume of work.
Apple iTunes Music Search Results Evaluation Project
Improving the relevance of search results for music-related content.
Raters must have access to an iOS device capable of running iOS 12.x. and higher. The job has an average volume of work.
Search Relevance task focused on improving the relevance of search results, based on the most likely intent of user searches. Raters must have access to an iOS device capable of running iOS 12.1.4. The job has an average volume of work.
Apple Store Search Relevance project focused on improving the relevance of search results based on user search intent.
Raters must have access to an iOS device capable of running iOS 12.1.4
Similar to Echo
Sharp
Apple Search Relevance task focused on improving the relevance of search results within different media categories.
Raters must have access to an iOS device capable of running iOS 12.1.4
Similar to Echo
Index
Apple App Store Search Results Evaluation Project
Search Relevance task focused on improving the relevance of search results based on user search intent.
Raters must have access to an iOS device capable of running iOS 12.1.4. The job has an average volume of work.
Similar to Echo
Slickrock
This project focuses on Facebook Stories and rating these personalized recommended stories.
Apple Safari Search Result Evaluation project. The goal of the project is to understand the user intention as represented by the query and determine how well or how completely the result satisfies the user's need.
Post categorization on Facebook. You have to decide what topic that post fits under. It is basically a categorizing system job which aims to put the posts that are made into the correct categories of broad topics.
Help improve ads quality for one of the world’s largest image sharing sites.
Required: An active Facebook and Instagram account – The Instagram account must be connected to your Facebook account. You also need a smartphone with Android or iOS. The job has an Above Average work volume.
There is a Facebook project called Tahoe that uses a team that checks events that come up in Facebook results. The main mission of this program is to bring people together through real-world activities by showing real events.
Facebook Falcon Projects. In this task, you will be asked to research and evaluate claims stated in various pieces of content (e.g., articles, links, photos, or videos). 20 hrs per week, work is done via Facebook rating tool.
The aim of this project is to help the client evaluate and improve the quality of the Ads that they present to their users.
Requirements are a personal computer running Windows 8 or better and a desire to look at different ads.
The aim of this project is to help the client evaluate and improve the quality of the Ads that they present to their users.
Requirements are a personal computer running Windows 8 or better and a desire to look at different ads.
Average volume of work.
No info
Tigris
The project consists of rating tasks focusing on improving the relevancy of advertisement results from a major social media platform. Consultants will rate advertisements to help the client understand which advertisements (also known as ads) people want to see, and why. The daily task of the consultant consists of rating a batch of ads in a client-owned rating tool.
FB project?
Carmel
Utterance annotation and evaluation.Microsoft Bing project aimed to learn search engine to recognize voice and human speech.
No info
*group of projects known as Falcon (mainly Facebook related projects) now is no longer actual, these projects also have been moved to Appen connect platform. Newly added projects such as Mohalwk, Prospect MI, Zeya, Copper and Alkali are most likely related to small Facebook testing projects. If you have been participating in one of the above listed projects, feel free to share your experience with us via leave a comment option!
Lionbridge
Lionbridge project code name
Description
Reference for more info
Internet Assessor, Personalized Internet Assessor, Rater
Looking to pass aPPEN Yukon project/lIONBRIDGE iNTERNET Assessor & Internet Safety Evaluator Exams? THEN YOU HAVE COME TO THE RIGHT PLACE FOR all THE INFORMATION THAT YOU NEED
(Updated Jan 2020)
Before you can start working on various tasks, you must pass the qualification exam. The qualification exam you will take covers two types of rating tasks which will make up 70% of all tasks in the project: Needs Met and Page Quality.
Once your candidature is approved for certification, you will get an exam email with detailed instructions. Within seven days you have to study 160 pages of the general guideline which consists of rating rules and concepts. The guide explains you how to do the rating of search results and assessment of quality of web pages. You should follow these rules and not use your personal judgment. What they want is for you to represent a typical user in your country. Before you proceed with the exam you need to study the guideline at least twice paying particular attention to practical task examples. Keep in mind that it’s 160 pages long. As I always advise, don’t wait till the last minute.
Note: At the stage of examination, Lionbridge provides candidates with very helpful exam preparation support. You will get access as an exam guest to the Lionbridge learning portal where you can listen to webinars and even try to do a few practical tasks before proceeding with actual exam.
Important update: Lionbridge is very selective in terms of recruiting new raters, up on revising exam results they will check your activity on their exam preparation portal, with this in mind, I strongly recommend you go through all the webinars on exam preparation portal before proceeding with the exam itself.
There are three parts in the exam: you can proceed with the next part only if you pass the previous one. Once you complete any part of exam, within 3-10 minutes you will receive an email with information about results.
Google updates the way the exam is done from time to time. The main feature that has been changed is that you must rate pages from your smartphone (send to device tasks). If you do not have smartphone, they say you cannot apply for this job. This is due to the fact that the number of mobile users is constantly growing and the company wants to assess websites from the perspective of on-the-go mobile users.
Note: There is a recommended schedule that will give you plenty of time so that you’re not rushing to complete the exam at the last minute:
Day 1 – Study the Guidelines – Make sure to setup your smartphone so that you are ready to take the final portion of the exam.
Day 2 – Start the Theory part of the exam making sure you are taking time to CAREFULLY read each question. Some of them can be tricky and that’s meant to make sure you fully understand the concepts.
Day 3 – Go back and study the guidelines once again starting with the Page Quality section.
Day 3/4 You can start the second part of the exam which is the Page Quality section if you feel comfortable on day 3. You may want to study this entire day, and you’ve got plenty of time if you decide this to take the exam on day 4.
Day 5 – Read the Needs Met section of the guidelines again and make sure to take your time. You’ve still got two more days to finish. And as it has been mentioned before, have your guidelines open in case you need to refer back to them.
Days 5/6/7 – Get started on the last part of the exam ‘Needs Met Rating’
Internet Assessor and Yukon projectexam part 1 (theory)
In this part of the exam you will be assessed on how well you understand the theoretical part of rating concepts, as well as on your ability to interpret questions and search for information in the general guidelines. It is an open book exam, so you can use the general guidelines to find answers to the test questions. This part has 32 question sections; each section has 4 parts, so in fact you have 132 true/false questions to answer. Read questions carefully, some of questions can be tricky in terms of their wording. You can consult with the guideline book while passing the exam and it’s strongly recommended to use this opportunity.
Note: Do not leave incomplete tasks more than for 1 hour, otherwise they will expire and you will fail the exam automatically. If you need to take a break, complete tasks and press “Submit and Stop rating”.
In most cases an hour would be enough to complete the theoretical part of the exam. The following message will be shown to you once you have submitted the last page of questions:
No tasks are currently available. Please try again later.
Within a few minutes a result notification email will come to your gmail, and in the case you are successful with part 1, the next part will be available in a few minutes minutes.
Lionbridge Internet Assessor and Appen Yukon projectexam part 2 (Page Quality)
In this part of the exam you will be asked to assess 7 web pages for their quality. My advice for this part is to check all quality aspects, and fill in the online form very carefully so you can give a correct overall quality score (from lowest to highest) to a page. You should expect 2-3 low quality web pages, 2 high quality web pages and 2-3 medium quality web pages. However this is not a rule, sometimes candidates get 4-5 pages of high or medium quality pages and only 1 or 2 of low quality pages to rate. Keep this in mind. In order to give correct ratings, examine web pages for ETA (Expertise, Authority and Trustworthiness) criteria very carefully.
Also, keep in mind that in this part of exam you need to give a PQ rating mainly to a web page and not to a website, although some quality aspects of a website have to be taken into consideration when assigning overall PQ rating, in particular in case of YMYL (Your Money Or Your Life) content which Google takes very serious.
Do not forget to write useful comments where required, otherwise the system may show an error.
You don’t need to finish this part at once. If you want, you can take a rest. The tasks are still available when you sign in. Again, if you pass this part, the company will send you an e-mail saying that you can proceed with part 3.
Note: The url/web page may not open on your smartphone, in this case do not mark these as “Did Not Load”. During the exam you should not expect to give a website this flag. If you do so, your answer will be incorrect. To solve this issue try to open link using incognito mode on your browser or/and give rating using other sources of information (website reputation, etc).
Page Quality Task
Internet Assessor and Yukon projectexam part 3 (needs met)
This is probably the most difficult part of the exam, and many candidates fail the exam at this stage. There are 7 tasks, and each task contains 8-12 result blocks. In each task you will be given a certain query, and a list of search results blocks with links where you have to give a usefulness rating to search results using a scale from “Fails to Met” to “Fully Met”.
Internet Assessor and Yukon projectexam Passing score
In order to pass this part you have to get an overall score of at least 80%.
YUKON EXAM PART 3 (NEEDS MET) SEND TO DEVICE – OPENING LINKS ON MOBILE & MULTIPLE Accounts LOGIN ISSUE
Note: while doing Yukon (Appen) and Internet Assessor (Lionbridge) exam part 3, keep in mind that all links have to be opened on your phone because Google analyses web pages from the point of view of mobile users. If you do not have a smartphone, you may use any android emulator for this purpose, in some cases it is even more convenient than using a smartphone for opening links.In order to open the link on your mobile phone, click the link on the result block on your desktop and then go to www.raterhub.com/url on your mobile browser (Google Chrome) to visit the link. You must be logged in your gmail account in order to have access to raterhub platform on your phone. The same rule works if you use android emulator.
Here is an official Leapforce (now Appen) video instruction on how to open links on mobile. Please note that this video was created by Leapforce and uploaded on Youtube by someone.
Note: Some candidates when accessing the link (www.raterhub.com/url ), got informed that Gmail account was unauthorized to visit the link. Most likely reason for this error is the fact of being logged on multiple Gmail accounts on a device. The solution for this problem is to go to android setting -> users & accounts -> add project associated Gmail account, then restart your device. This setting should fix the issue.
Lionbridge Internet Assessor and Appen Yukon projectexam part 3 (Needs Met) Expert Tips and recommendations
Additionally I would like to give general tips on how to proceed with Yukon (Appen) and Internet Assessor (Lionbridge) exam part 3:
1. Make sure that you fully understand the query and user intent. If you do not fully understand user intent, make additional research on the Internet.
2. When you are evaluating landing pages in the exam, don’t place too much importance on the Page Quality slider. The focus needs to be on Needs Met Ratings. Do not worry about using flags in the exam such as the ‘Upsetting-Offensive Content’ flags.
3. One of the most important things is to absolutely know when to use Fully Meets. Fully Meets ratings are very rare compared to Highly Meets ratings. Avoid being too generous on landing pages. If you have any doubts at all, you should not be rating a page ‘Fully Meets’.
4. Determine what type of query (user intent) is:
o Single entity (business) query with local or non local or both intent/to visit in person/navigate to a website
o Business chain query or query which is looking for business within one category (e.g. car repair service near me)
o Simple know query (where the user requests a specific piece of information, do not assign the Fully Meets rating to any particular result, the correct answer must be present in an SCRB).Examples of simple know queries:paris weather, delta airlines customer service number, current time in Amsterdam)
o Information query seeking a long answer/piece of information (what cases stress)
Types of SCRBs (Special Content Result Blocks): ny weather (simple know query), how old is universe (simple know query), does god exist (long answer seeking), berlin airport (single entity), burger king amsterdam (chain), coffee shops amsterdam (categorical).
o URL/website query
5. Consider the following aspects while giving ratings to result blocks: user location, for local intent queries far away results are useless or slightly relevant. This aspect is very important for the following types of queries: tire shops near me, where to eat near me, Walmart near me, etc. When you get a task with this type of query do use in built map for researching user location/view port and results.
6. Make sure you give correct ratings to special content result blocks for single entity and business category queries. In case of business chain/category query, the result block with a list of businesses is always better than just result for a single business within one category.
7. Consider expertise and trustworthiness of search results for financial, legal, and health related queries; low expertise and trustworthiness automatically reduce needs met rating of the results.
6. Think of dominant, possible common, and minor interpretations for the query and take this aspect into account while giving ratings to a result block. Minor interpretation results should get a low usefulness (Needs Met) rating.
8. Take into account how well the web page/website is optimized for mobile users.
9. Once you have analyzed and considered all listed above aspects, you should give a rating to result blocks one by one.
Note: In some cases there might be some results (links) which are not accessible from outside U.S. (usually websites of well known retailers such as Target, Walmart, etc. They ban access from foreign IPs to prevent hacker attacks). Appen, Raterlabs or Lionbridge do not take into account this factor and therefore rating such results as fails to met may lead to overall fail.
To make sure you do not miss anything before proceeding with part 3, follow pre-exam instructions to pass this part. There will be a link to a video. Watch it and follow the instructions.
Examination Needs Met Task
Note: In part 3 of the exam (Needs Met), most of results blocks have Needs Met and E-A-T (previously PQ) sliders. Main focus should be on providing correct Needs Met rating, which in some cases might be affected by E-T-A (PQ) rating when it comes to YMYL queries and results. Leapforce/Google re-named PQ to E-T-A in part 3 to make it clear that the most important aspects in PQ are expertise, trustworthiness and authority. In general it is similar to PQ concept. While giving E-T-A rating you should evaluate MC under these 3 criteria. In most cases you may do it fast at a first glance approach, but be sure to provide good evaluation for E-T-A when you have YMYL query.
Note: If the user location map does not show user location correctly, and you have a blank page, hit the black arrow button. The task will be closed, but you will see “Incomplete tasks” on the evaluation platform. Click continue and come back to your task with the updated user location field. Remember that it is very important to take into account the user location while giving ratings to block results. Also keep in mind that there is no need to finish the task in one sitting. You can take a break if you need to.
Once you complete 7 tasks you will get this message:
No tasks are currently available. Please try again later.
Yukon (Appen) exam results and Internet Assessor (Lionbridge) exam results
You will get overall exam results on the exam due date from Appen and exam due date plus one day if your vendor is Lionbridge. Appen will ask you to sign a few electronic forms and in 2-4 days you will be given access to live tasks. At Lionbridge you will have to complete educational course with no pass/no fail quiz before you are granted full access to the tasks.
US YUKON Raterlabs and Lionbridge raters
Once you pass the exam you will then go through the employment process which might take up to 3 weeks. You will be asked to complete an online version of the IRS Form 8850 (Pre-Screening Notice and Certification Request for the Work Opportunity Credit), and a tax credit questionnaire from a vendor payroll provider. After you have completed these two forms, you will be presented with a formal employment offer letter. After acknowledging your offer, you will still need to complete the required activities in a timely manner. Try not to postpone the completion of these important steps: Providing bank details for ACH payments on Paycom website. Providing your I9 Employee Section 1 and I9 Employment Eligibility Documents (section 2) must be completed by specified deadline. It is very important that candidates complete all required steps as soon as possible to avoid termination of your employment.
NEW!!! Welcome to Search Engine Evaluation Exam (Yukon, Internet Assessor) Simulation Tool!
What is The Exam Simulation Tool?
I have created a special tool that will help anyone who is getting ready or considering taking the exam. This tool was designed to assist candidates with exam preparation with the main focus on practical rating tasks that include Page Quality, and Search Results Satisfaction ratings. The tool has 5 Page Quality tasks, and 5 Search Results Satisfaction (Needs Met) tasks of all the major query types. (If you’re not sure what a query type is, that is the keyword that a user searches Google for). These include; simple-know, local search, information search, website/navigation, visit-in-person/business which cover basically all rating concepts/tasks types and skills which will be tested during a real exam.
How can this tool can help you pass the exam?
The way the tool is designed will show you exactly your problem area and what you need to focus on. It helps you identify your level of understanding of the rating concepts, and your ability to apply them by practicing first, and then identifying your weak points. You will receive your overall score (pass or fail), detailed feedback on your mistakes and know exactly what you need to go back and study in the guide. Of course we need to mention that there are no real actual exam tasks because that isn’t going to help you understand anyway. Each one of these is hand designed from the most typical exam tasks, and is also designed to be the best simulation possible. If you pass this, you will be ready to pass the actual exam.
How do I get my hands on the exam simulation tool?
If you’re still skeptical, we have a free demo version which you can try FREE right now! It contains a simplified Needs Met and one Page Quality task so you can see how it works. Once again, the demo is 100% free and remain completely anonymous.
The tool has 5 Page Quality tasks, and 5 Search Results Satisfaction (Needs Met) of all the major query types. These include; simple-know, local search, information search, website/navigation, visit-in-person/business which cover basically all rating concepts/tasks types and skills which will be tested during a real exam.
If you are not confident about going through the Appen Raterlabs or Lionbridge qualification exam, you may always get my expert support and advice. For detailed information contact me here.
Payoneer Payment method for Appen
Leapforce/Appen/Raterlabs candidate? If you get a job at one of these places, you’re going to need Payoneer account because that is the only payment option available for these companies. Simply sign up for Payoneer under my referral link and you’ll get the simulation tool subscription fee back as Payoneer subscription 25$ bonus once your account is loaded with 1000$! Normally it takes 1-2 months to make 1000$ for EU/USA/Canada raters and 2-4 months for raters from another countries.
I have created an exam simulation tool that will help anyone who is getting ready or considering taking the exam. This tool was designed to assist candidates with exam preparation with the main focus on practical rating tasks that include Page Quality, and Search Results Satisfaction ratings. The tool has 5 Page Quality tasks, and 5 Search Results Satisfaction (Needs Met) tasks of all the major query types. (If you’re not sure what a query type is, that is the keyword that a user searches Google for). These include; simple-know, local search, information search, website/navigation, visit-in-person/business which cover basically all rating concepts/tasks types and skills which will be tested during a real exam.
How can this tool can help you pass the exam?
The way the tool is designed will show you exactly your problem area and what you need to focus on. It helps you identify your level of understanding of the rating concepts, and your ability to apply them by practicing first, and then identifying your weak points. You will receive your overall score (pass or fail), detailed feedback on your mistakes and know exactly what you need to go back and study in the guide. Of course we need to mention that there are no real actual exam tasks because that isn’t going to help you understand anyway. Each one of these is hand designed from the most typical exam tasks, and is also designed to be the best simulation possible. If you pass this, you will be ready to pass the actual exam.
How do I get my hands on the exam simulation tool?
If you’re still skeptical, we have a free demo version which you can try FREE right now! It contains a simplified Needs Met and one Page Quality task so you can see how it works. Once again, the demo is 100% free and remain completely anonymous.
The tool has 5 Page Quality tasks, and 5 Search Results Satisfaction (Needs Met) of all the major query types. These include; simple-know, local search, information search, website/navigation, visit-in-person/business which cover basically all rating concepts/tasks types and skills which will be tested during a real exam.
Want to Apply For Search Engine Rater and Social Media Evaluations jobs at Lionbridge and Appen?
Sometimes finding these types of jobs online takes a bit of research. Most of the big companies that offer these types of jobs don’t usually advertise in common places or online job boards. You usually have to go directly to the company and apply on their website. I have seen that Appen Global advertises their jobs on Flexjobs.com, but that website charges a monthly fee. I never quite understood how a company can charge you to find a job, but they are the exception.
Finding these jobs and applying for them are different for each company. Going to each individual website is the best way, as long as you know where to go, and how to apply. With that being said, this article, and the videos we have provided, will show you how to find the work from home jobs.
One of the other important things to consider when looking for these types of jobs is your location. Some of the contracts that are offered through companies like Appen and Lionbridge require you to live in the demographic area that the project requires itself. The reason for this is because the data from different regions will be different depending on location and the language.
Before you begin to apply for these jobs, or really any job for that matter, it’s probably a good idea to go over your resume and make sure it’s up to date, and that it includes only the important information that is necessary. It might also be to your benefit to have your resume checked by a professional. The resume experts at Resumewriters.com is an awesome service that I have used in the past, and having your resume edited or corrected by a professional is a sure fire way to get noticed and get you the job you want. Many of our readers have had very good results with them.
To visit their website Click Here: Resumewriters.com
Another important note is that if you have worked previously on other social media projects with different companies, don’t mention this on your CV/resume. The two most important factors about this are that you do not need previous experience for these jobs. And second, if you have worked on these types of projects before you will most certainly be looked over and turned down for similar projects. The reason for this is uncertain, but we have seen other people turned down because of this.
You can work on two different projects for more than one company. However, you need to use a different IP address for one or the other. So if you work for Appen and want to work on another project with Lionbridge, you will likely need a different IP address. Appen seems to be the only company that records an IP address and this is shown in the backend or the portal of the work dashboard. This is why it’s a good idea to use a separate IP address for another project. It might also be a good idea to use a different email between companies when applying for and working on these jobs.
Some advice when working on different projects so that you don’t overwhelm yourself is to get some experience working on one project with a company before trying to get another contract with a separate company. Most of these contracts are part time. So its beneficial to have more than one project at a time if you can handle it.
A really important thing to note here is that you can only have one contract per IP address when it comes to these companies. So if more than one person is working for any of these companies, it is important to have a separate IP address. You can usually order another IP address from your internet service provider for about $4-5 per month extra. Check with your ISP to find out if you require more than 1 IP address per household. You can read more about this in FAQ
Application process at Lionbridge – Ads Assessor, Internet Safety Evaluator, Ads Assessor
Lionbridge search rater project is called the Google rating project (ads assessor, internet safety evaluator, ads assessor) which is the same as Yukon at Appen (Raterlabs). They also have the positions of Personalized Social Media Assessor, Personalized Internet Assessor, and Personalized Web Assessor.
Application process at Appen – Search Engine Evalutor, Rater, Internet Analyst
At Appen/Raterlabs you would apply for search engine evaluator position first, and then based on project availability, you would get invites for Yukon, Nile, (Facebook social media evaluator) Truckee or Shasta projects. You can work on more than one project at a time.
Appen Global
Many of my readers will be interested in how to get invited to other projects at Appen Global. Appen Global offers a lot of different types of projects that you are able to work on. Unfortunately, is not likely that you would be contracted to start on projects such as the Thunderbird music project from the beginning. You would be more likely to start on a larger project first such FalconInstaAds or the Arrow project. Once you have started working at Appen, and you are on a Falcon project, like any of the social media projects that they offer, you will likely get an invitation to start on other projects as well. The more you work, and the better your performance is, the more offers you will get from Appen.
Those projects include:
Arrow (ads assessor);
Yukon (search engine evaluator);
Nile/Truckee Facebook/Instagram ads assessor projects (social media evaluator).
application process video instructions -Lionbridge, Appen, Appen Global
We thought it would be a good idea to create some quick walk-through videos to show you how to apply for these jobs and the different companies. Some of these:
Lionbridge, go to http://www.thesmartcrowd.com/workers/job-opportunities/country/russia/#jobs-list
Appen, go to https://connect.appen.com/qrp/public/jobs/list
Appen Global go here: https://talent-appen.icims.com/jobs/intro
How long does take to hear back from Lionbridge and Appen?
It depends on many factors, sometimes immediately, and sometimes it can take 3-4 months.
It might be that at the moment of your application submission, quota of raters for your country was reached and therefore Appen or Lionbridge does not hire new rater at the moment. As soon as they fire unsuccessful raters, they continue hiring new people to maintain the required number of raters in your country.
Need to use the Payoneed Payment Method for one of these vendors? Payoneer is a payment processing company like PayPal, or Skrill. Payoneer main focus is on providing payment services for freelances, small and medium scale e-commerce business and international companies like Appen Global.
All Payoneer customers have two options to choose how money can be utilize from Payoneer account where all payments come in:
Withdraw payments from your Payoneer account to your local bank account, in local currency:
This option is useful for those customers who have a local bank account and would not like to get one more bank card. You simply withdraw money from your Payoneer account to your local bank account.
The second option is pre-paid card or deposit bank card:
Benefits of pre-paid card:
Non US residents can open a bank account in a US bank and get the prepaid master card in USD currency.
Possibility to purchase goods in US online retailers for non US residents such NewEgg, Amazon, etc.
You can withdraw funds anywhere in the world
Fair service and conversion fees
Possibility to withdraw money in USD currency from PayPal to your Payoneer prepaid card without currency exchange in PayPal system
Payoneer as a payment method for Appen
Appen Global was using one of the largest payment processors ‘ADP’ but in 2017 Appen decided to use Payoneer for making the majority of their payments. It is much easier to get the account setup and have this beforehand. As an independent contractor, you are always responsible for doing your own taxes, so Payoneer will not withhold taxes. You must do this on your own. Payoneer will make you go through some approval process but if you are working for Appen Global, you will be approved as there is no other way to accept payments. Even if you may not use it for Appen (if you do not into any project) you can use it send and receive payments for majority of services/websites as it is becoming an alternative for PayPal.
Payoneer now has a sign up program that’s a good incentive to join, let’s earn 25$
If you get a job at one of these places, you’re going to need Payoneer account because that is the only payment option available for these companies. Simply sign up for Payoneer under my referral link and you’ll get the simulation tool subscription fee back as Payoneer subscription 25$ bonus once your account is loaded with 1000$! Normally it takes 1-2 months to make 1000$ for EU/USA/Canada raters and 2-3 months for raters from another countries.
EXCLUSIVE PAYONEER SIGNUP OFFER
Get $250 commission once referred user reaches a unique load of $5,000 within 3 months time. Make sure you can meet required conditions before signing up for this exclusive offer via this Payoneer signup landing page.
How to Sign UP to Payoneer
HOW to add Already existing Payoneer account at appen connect personal project page – how to link appen global to pyoneer
After successful completion of qualification process at Appen you will receive an email from Appen financial team (Register to Payoneer).
As part of our ongoing quest to streamline our services, we have partnered with Payoneer, our payment provider that will process your approved invoice payments on our behalf. This means you will need to setup an account with Payoneer by visiting your personal account registration page located in your Appen account located here: https://connect.appen.com/qrp/core/login
You will not be able to submit invoices using our online invoice system until after you have successfully linked your Payoneer account using the provided registration page.
Make sure that you sign up to Payoneer under my referral link to be eligible for 25$ bonus.
After completing Payoneer signing up process, login into your appen connect project page (https://connect.appen.com/qrp/core/login)
Click on Begin Registration. You will be directed to an online application page, where you can choose to get paid to your already existing Payoneer account.
Click Already have a Payoneer account?
Enter your Payoneer username and password and follow the on-screen instructions. That’s it, you’re connected!
How To Add Payoneer For US raters at Raterlabs and Lionbridge (Direct Deposit)
Payoneer offers receiving accounts via the Global Payment Service, which enable you to receive local bank transfers from companies and marketplaces in the US, Europe, UK, Japan and China directly to your Payoneer account.
US Lionbridge independent agents and part-time raters, Raterlabs part time workers may use Payoneer account for receiving payment from Raterlabs and Lionbridge. Only US raters can use this method of payment since only ACH (US local bank) transfers in USD can be accepted. Raterlabs and Lionbridge send payments to US raters via ACH (direct deposit), while international raters revive their payment via wire transfer (Lionbridge).
Where to find your Payoneer bank details to provide Raterlabs or Lionbridge with?
Go to your account page->Receive->Global Payment Service
You will then see the bank information page here including name of bank, your count number, and the bank routing number which you will need to give to Raterlabs or Lionbridge when setting up direct deposit:
This is a review of a new project out there called Point Of Interest, or for the sake of simplicity, we’ll call it the POI Project.
As with all projects, this one is different in the kinds of tasks being performed. The main goal is to evaluate the quality of POI data as a whole, and we’ll explain what that means later on.
The term POI generally refers to any entity that you would find in the Yellow Pages directory like a mall, a movie theatre, or some kind of department store. These are places that users want to find on a map and potentially navigate to it, so you’ll have to check and make sure that they are correct and accurate as possible. We probably don’t need to go over ALL the examples of POIs but it can be anything from a business to a farmers market.
However, there can also be a POI in a POI such as a department store in a mall. Examples of some other Parent POIs include, hospitals, universities, parks and amusement parks, hotels, golf courses, beaches, and even boardwalks can be a point of interest.
Each point of interest is associated with additional information that helps a user locate or contact a point of interest. This is called the POI listing information and what a user sees on a POI info card when interacting with the POI on the map. The accuracy of the listing information must be verified and we will go over how you would do this.
Each POI has the additional information that you’re going to need to accurately verify the listing including:
• Name
• Category
• Address
• Phone number
• Hours
• Official URL/website
• Pin location
• Entry points
When doing an evaluation, you are presented with three different types of questions.
POI Validity – This is pretty straightforward and asks: Does the POI represent an actual entity? So if you are shown a place on a map where an entity should be, but there is nothing there, you would answer NO to this question. However, chances are that most of the time you will answer YES to this question.
Data Accuracy – Is the POI Information Correct? You’ll want to take a look at the info card that we just went over and determine the accuracy of each opponent of the POI listing information. If the POI doesn’t meet the requirements to be considered ‘valid’ then you do not need to verify the listing.
Comments – Are there any issues with the POI?: This is not really a question but you are presented with a comments box that must be filled out whenever conflicting information for a listing is found. Your comments should include links to any websites that you used to verify this information or other information that will be helpful for future visits for that POI.
And that is where some good research skills are going to come in handy. You are EXPECTED to do online research to verify the validity of the POI and the listing information. You can do this by using the official web pages, social media, street imagery, and postal authorities.
If you cannot find official resources then you should see if there are other reliable sources such as newspaper sites or crowdsourced user review sites. Use your search engine skills to locate official resources and try to find some local knowledge when relevant to understand POI details.
The rating interface is not bad looking and you shouldn’t have any problem quickly marking what you need to using the drop-down menus and selecting the appropriate result.
Most of the time you will be responding Yes to this question unless the entity doesn’t appear at all on the map. If you do answer Yes then additional Data Accuracy Questions will appear.
If for some reason the Address Accuracy Question is rated as ‘Incorrect’ then you will see a checkbox list of address components appear and it will look like this:
We should point out that the POIs evaluation requires research to verify the POIs name, category, address, phone number, hours of operation, the official website location including the latitude and longitude (which is displayed as a PIN on the map) and entry points to the location or entity.
We’ve already gone over what types of resources you should use to verify this information, but we will quickly go over it one more time. Make sure you perform a regular web search of the POI by the name and the town, city, or village to see if an official resource, such as a website, can be found. Only by doing this can you judge if a POI is valid or not.
Official websites will definitely be the first place to start and here we have provided a few examples.
There can be other official resources as well even if the entity doesn’t have its own website. Make sure to check social media sites that are claimed by the business and have some updates within the last six months. Street imagery for the business showing streets or buildings where the entity is. Remember, you are trying to see if this is a real place that still exists or not.
Postal authorities can also help with postal codes or locality names if the entity exists.
If the POI lacks an official website, then officially managed social media pages can also be used to validate the information.
However, when doing this job, there are some online research exceptions. These include EV Chargers such as PlugShare or Alternative Fuels Data Center, Bike Share POIs which use at least three different street images, bus stops which may or may not be obvious, and in-store kiosks.
POI Validity
There are times when you will not be able to validate a POI listing. These could be that the entity has closed down, it could operate from a residence and be registered to a residential address. There are also Seasonal POIs such as a farmers market or a seasonal amusement park. There are many examples of these in the official guidelines which should be kept handy at all times while validating. Here we provide a few examples to give you an idea.
If a POI has closed or shut down, it might be really hard to identify but there are some things you can do. Chain POIs that have different branches will usually list those locations on their official site and you can also check social media to see if people are saying the entity closed down. Check the street imagery of where the entity should be. If you don’t see it, you’ll mark No – Closed.
There are some other exceptions when it comes to research on this that can be found in the guidelines including tons of exhaustive examples of POI Types and how they should be rated including an explanation of why it was rated the way it was.
As we always say with every project we review, make sure you read the guidelines over and over until you are certain that you understand what you are doing and how to do it.
Name Accuracy
Obviously, it’s important that the name of the entity is correct and straightforward. If you see any slight alteration to the name, then it will be rated as partially correct.
There are three different options in rating the name as follows:
URL
This is easy to understand. Simply look at the URL of the website and see if it’s correct or not. If it’s not, then you will rate it depending on which option fits it best.
The guidelines have many exhaustive examples when it comes to URL Rating examples and shows you how to decide if the URL is correct, or if it’s not.
Address Accuracy
Every entity should have an address associated with it so that users can navigate to the business or entity. There are two different address types: Formatted address for displaying to users and the Structured Address is the ‘raw’ version of the address used in the database.
POIs within POIs
We touched on this a little bit at the beginning of our review. These are businesses that are inside of a parent entity such as a mall, hospitals, universities, train stations, and department stores. Keep in mind that a phone number of a POI within a POI that is listed on the info card must meet two things before it can be called ‘Correct’. Check the guidelines for more information on this.
Hours
POI hours need to be verified using official resources only and you’ll see the following Hours Rating Options.
Remember that there will also be hours that are different for the child and parent entity. For example, a mall may stay open while other shops inside close early. You’ll need to make sure that the hours listed are the child entity for which you are researching and not the parent entity.
Some POIs also have seasonal hours and the guidelines go over all these with a lot of examples.
Category
Categories for POIs got to be correct as well and you need to determine if the category is accurate and the best fit available in the category list. Some examples of category ratings are as follows:
There are many different examples in the guidelines to help you decide which category the entity should be rated at.
If you have ever looked at a Google Map for an entity, then you know that there is a Pin that is supposed to mark that location with a certain accuracy. So for each POI, you would check to see if the pin is exactly over the POI or if it’s not accurate and placed somewhere else. The following are the Pin Rating Options:
There are literally 11 or so pages on this topic alone in the guidelines as there can be many different options or exceptions when it comes to checking the accuracy of the pin.
Entry Point Accuracy
It’s quite obvious what an entry point is, this is when you would be told by Google Maps that you have arrived. That said, there is more than just one entry point including Vehicular Entry Points, Pedestrian Drop Off Zones, and Pedestrian Entry Points such as the front doors to a large mall or hospital. The following are the Entry Point Rating Options
Again, the guidelines give some exhaustive examples of this and how to rate the entity including when there are no entry points available at all including a helpful flowchart that helps to make a decision on Entry Points.
Comments
The last thing is the comments. Comments are incredibly important because here you will be able to list all the issues you might have had in researching the entity. It is important to understand that comments are MANDATORY anytime you find conflicting information and it needs to be corrected. Some of those issues might be as follows:
Make sure that when you comment that you refer to the 150-page guidelines to help you understand how to reference those issues that might arise. Make sure that you are concise, and skip being too elaborate stating only the problem. Also, English is the only language that comments can be written in.
We hope you found this information useful, and we would like to hear from anyone who is working on this project. As always, if you need help passing an exam, have some questions about one of these work at home opportunities, or if you need any help at all, please leave a comment or email us and we will reply as soon as possible.
Welcome to this informational review of the Slickrock which is a project that focuses on Facebook Stories and rating these personalized recommended stories.
Stories are defined as temporary content that is created by a page or a user. Stories on Facebook automatically disappear after 24 hours and can only be viewed during this time and it disappears after this. The important thing to know is that a story can have several pieces of content and include multiple photos and videos.
Stories can be found in the ‘Stories Tray’ which is located at the top of the newsfeed for those who are not familiar with how to view them and what you’ll be looking at if you decide to try for a job as a rater in this project.
The reason that human data collection is needed for the Stories Recommendations product is so that some users will see stories from people they don’t follow, but may be interesting and enjoyable.
Feedback from raters is used to determine whether stories that are recommended to some users are actually relevant to them. Each rater is given a set of personalized recommended stories that Facebook thinks will be interesting to the user. You would then rate the relevance on each story for that particular user.
Fecebook Rating Tool and the rating process
There is an online user interface that each rater logs into that is provided by Workplace by Facebook so the tools are directly made by them. You simply log into the system and then choose either Assign 20 jobs or Just Go which will do the same thing and give you random stories to rate. These will give YOU stories to rate as how you feel about them. More on that later…
Once you are logged in this is what you would see when you begin rating:
When you see a story that has multiple pieces of content, you will need to answer some questions based on what you see in the stories.
So now you have you a sneak peek at what the interface looks like, we’ll tell you a little about how you would rate recommended stories. As we said previously, these are stories that are presented to you that are likely based on your likes and preferences including your interactions on Facebook. The goal of the project is to help FB understand as best as possible how you felt about each individual story that was recommended to you.
You need to ask yourself the following questions:
Question 1: Did the recommended story make you feel uncomfortable or was it unacceptable? Some of those could be from these examples below:
Nudity/partial nudity
Violence
Political views
Religion
Feeling spammy or inauthentic like an ad
Being in a language that you are not familiar with
In these instances you would click the ‘Yes’ option and then explain in your own words why the content made you feel the way it did.
There will come a time when you will see stories that are based on public figures and you will be asked how familiar you are with them and asked if you recognize them in the thumbnail or the story itself. There are different levels of familiarity which range from Very Familiar where you know a public figure reasonably well such as a performer like Lady Gaga, or someone else equally popular, and you are interested in them and their activities.
Mildly Familiar is when you have heard of the person but are not deeply into their activities or what they do such as knowing the name of a sports player like Lebron James but not knowing what sport they play.
And at last there is the Not Familiar and you have no idea who the public figure is and you have never seen them before.
Therefore, knowing these you would select one of these three options depending on how well you know the public figure.
The third question asks if the author of this recommended story is from your area of interest. It is perfectly fine if you don’t know the person as long as they fall under your areas of interest. If you do not recognize the author, then you would go to their page and look to see if what they are about is relevant to you or not. If the author falls under your area of interest, then you select YES, but if not then NO.
The fourth question wants to know if the content of the story is interesting to you or not. Don’t forget that a story can contain more than one piece of content or video. It is VERY important to understand that you need to define the content interesting to you even if there is ONLY ONE piece of content in the recommended story. If just ONE story is interesting to you then you would mark yes. However, if none of the recommended stories interest you, then you should select no, and then you need to explain in your own words why none of the stories caught your interest. It is important to explain as simply as possible, and give as much useful information as possible and not answers like ‘I just didn’t like it.’. There is an option to Skip or Reroute but these should never be used unless there is a problem with the content not rendering.
Once you are done answering all four questions, you would then click on ‘Submit Decision’ which will then take you to the next story. Each session to rate is 30 minutes long.
If you are working on this project, we’d like to hear from you in the comments. If you have any questions on this project, or if you need help passing any exams related to getting any jobs we have listed here, leave a comment or email me and we’ll get back to you as soon as possible!
There is a Facebook project called Tahoe that uses a team that checks events that come up in Facebook results. The main mission of this program is to bring people together through real-world activities by showing real events. This is done by working with tools and resources that allow users to discover interesting real events that they might be interested in attending.
TAHOE PROJECT QUALIFICATION PROCESS
To qualify for this project, you must take a quiz that has 60 questions. 53 of the questions are practical, and 7 are theoretical. The exam looks like it might be difficult to pass, and you must have a score of 90% in order to pass it. Be sure that you follow the guidelines and the instructions in order to make sure you are giving the correct ratings. If you fail the exam, it can be retaken. However, if you fail the exam the second time, chances are they will not give you another opportunity. Get Expert Personalized Assistance with Any Project Qualification
Tahoe project rating process and logic
Improving the discovery of events is done by making a distinction between real events that are happening in the real world at a specific time, and Non-Real Events which are obviously the opposite and not happening in the real world. The job of the individual working on this team is to label events as either Real or Non-Real events by reading the event and basic information. If needed, the Facebook page is opened, and the page is rated either a real or non-real event. This data will be used to help train an algorithm to be more accurate when it comes to events that are shown on Facebook. Non-real events are not removed from Facebook, so this work is important to help people find real events that are happening in their area.
When looking at an event, the main question is to ask if the event is an activity in the real world, and if it is, is there sufficient information for the user to attend the event like date and time, and directions or a phone number to reach someone. Chances are if it has all this information on an event, then it’s a real event and can be tagged as such.
The location for the real event should have the city, state, or a venue where the event is listed and usually found in the description box.
Time is important of course and there are some rules as far as the time is concerned.
Events that last up to 3 days from start to end, such as weekend festivals, are acceptable. Events that have a duration of more than 4 days are only considered an event if there is enough activity within each day to justify having one event created for the entire duration such as a retreat event.
When looking at these event ads, it is important to watch for phrases such as “time to be determined at a later date” which tells you that the event creator doesn’t have a set date, therefore the event should be tagged as Non-real.
Reoccurring events and long-term exhibitions need to have event pages for each day those events are going on. Some examples of this are a movie theatre that has midnight showings every Friday. This needs an event page for every Friday. An art exhibit that is being displayed at a studio for a month needs an event page for every day it is being displayed. This also goes for several performances on the same day. Each performance is considered a separate event and should have a new page.
The activity of the event doesn’t need to be explicitly stated in the description. Keep in mind that the main intent of the event post does not need to be about the activity itself, such as a page selling tickets to an event. This is still considered a real event because its selling tickets to a real event.
The key here is to watch out for events that closely resemble an advertisement rather than a real activity for the user. This includes things like promoting sales, tour packages, causes, etc. These are not considered real events.
What is considered a Non-Real event?
Depending on the location, many events will be encountered that initially seem like they are real events, but in reality, they do not satisfy the event’s realness requirements. When evaluating these types of posts, remember to always use the main frame of thinking as with every post: Is this an event I can attend, and if it is, is there enough information for me to attend the event?
Below are some examples of some common Non-events that will be encountered.
Promotional events make up the majority of Non-Real cases that the rater will encounter during this experiment. Most of the time these events will more closely resemble advertisements more than actual activities or events such as a company using the event creation tool to advertise a discount or a sale. Travel agencies creating an event for the purpose of selling some tour package. However, if it is an experience type of event or an educational event such as an ecological walk, even if its called a tour, if its still a real event that holds interest for locals then it is considered real. One key thing to note is that these posts are usually made by non-profit organizations, government agencies, or educators and are a good indicator that they are real.
Watch out for insufficient times such as an event that has a start and end time that doesn’t align with the actual activity for the event, although these events are very rare.
Time details to be determined is another factor in determining if something is Non-real. If there is no strict date, then its nothing you can plan for, and therefore not a real event.
If you see a post that has a schedule of events, it needs to be tagged as non-real as every event should have a single post of its own.
Call to action events are created for the purpose of receiving donations, votes, or any other type of contribution from a user without the need for in-person activity. With that said, if the event is a fundraiser that occurs at a real-world location, this would be acceptable as Real.
If the event post is gathering support to see a certain performer or artist in a specific city or country, then the post should be considered non-real because there is no confirmed real-world event.
Online events are considered Non-Real because they do not happen in the real world and that includes live streams, radio, and TV shows. Watch out for memes, just for fun, and spam that pretends to show real events happening in the world, but upon inspection, you realize that it’s evident that it can’t be real such as bands performing in unrealistic situations, or events that are centered around popular internet memes.
Another thing that event creators like to do is use the event creation tool to announce a large-scale celebration or holiday. If it’s just some announcement that doesn’t lead to a specific activity, then it gets labeled Non-real.
Below are some more examples:
Jobs should be rejected if you do not understand the language that the post is in. Any post with solicitation of Child Exploitation or Child Nudity should be escalated immediately with the JOB ID to the manager. If you are uncomfortable reviewing the content, then the job can be skipped after it is escalated. Canceled events are still marked real as long as they fulfill the Real event requirements.
The language of the experiment should be present in the event title or description. If these are missing then you can immediately reject the job. Make sure that time details listed for an event make sense for the actual activity, and also make sure that the city and state, or venue name is listed somewhere in the event info.
If you work on this project or know someone that does, we would like to hear from you in the comments about your experience on this project. If you have any more questions, then shoot me an email or just leave a comment and I will answer as quickly as possible.
There’s a new job position that is available through Sykes for a Search Quality Evaluator working from home part-time.
It isn’t known exactly what vendor this job could be for since Sykes is an outsourcing provider, but it could be Sykes, Appen or even Lionbridge. Another thing to note is that this job was posted on December 21, 2019 and is still listed. So, if you are looking for a job as a Search Quality Evaluator (known as Google Search Engine Evaluation project: Yukon, Internet Assessor, Rater) this could be your big work at home opportunity.
Judging from the job description about searching for online topics, and how frustrating it can be to get bad results, this is most certainly the same type of job as Search Engine Evaluator and you would be responsible for performing simple web searches on specific topics and then rating the results that are returned within the time frame they give you. A Windows PC is required for this work at home job, and no MACS are being accepted in this project. You must also have an activated smartphone with Wi-Fi capability. Applicants need to have strong computer skills or they will not be considered.
The time you want to work on this project is flexible and you can work on your own time as you wish. There is a schedule provided after training, although it is uncertain what this training might be since there is no previous training for these types of jobs. Only the guide is supplied to you and you must learn on your own. After this training period you must work a minimum of 20 hours per week.
One thing to keep in mind for this project is that you would be reviewing offensive and mature content. If this type of stuff offends you, then you may want to skip this one.
Also, if you live in these states, Sykes is unable to hire you: Alaska, Arizona, California, Colorado, District of Columbia, Hawaii, Illinois, Iowa, Maine, Massachusetts, Montana, New Hampshire, New Mexico, New York, North Dakota, Ohio, Oregon, Rhode Island, South Dakota, Vermont, Washington and West Virginia.
You may apply for Search Quality Evaluator role here:https://jobs.sykes.com/en-US/search?keywords=Search%20Quality%20Evaluator&location=
There is no mention of what the pay is at this time. If you happen to be working on this program, we’d like to hear from you about the training procedure and what the rate of pay might be. As always, if you have questions about this project, send me an email or leave a comment and I will reply as quickly as possible.
Hourly rates across different countries: Lionbridge, Appen
Important Note: Some Appen Global projects are not currently available in many states in U.S.A. because the payment rate Appen Global offers is below the minimum wage in some of these areas. The company Sykes is also experiencing this same problem. Currently, the only alternative is Lionbridge. Our sources show that in many countries Appen Global pays 2 times less than Lionbridge, and to maximize profit, they cut wages. For example: In India Lionbridge pays $7 per hour, Appen $3 dollars an hour compared to U.S.A. where Appen Pays $10 per hour and Lionbridge Pays $14.
Country
Vendor/Project
USA
EU
Turkey
Brazil
Japan
Russia
Appen
Echo, Index, Sharp, Cable
12$-13,5$
14$
7$
Arrow (Google)
10$
14$
was 8$/now 5$
was 8$/now 5$
15$-17$
was 8$/now 5$
Truckee (Insta)
was 13,5$/now 10$
14$
was 9,45$/now 7$
was 9,45$/now 7$
was 8$/now 5$
Nile (Facebook)
was 13,5$/now 10$
14$
was 9,45$/now 7$
was 9,45$/now 7$
15$
Shasta
per task
per task
per task
per task
per task
per task
Yukon (Google)
was 12,5$ (net)/now 10$ (net)
14$
was 10,5$/now 5$
was 10,5$/now 7$
14$
was 8$/now 5$
Lionbridge
Internet Assessor, Rater, Internet Safety Evaluator (Google)
14,5$ (gross)
14$
10$
12$
7,3$
Ads Assessor (Google)
13,5$
14$
10$
12$
7,3$
Maps Analyst (Apple)
per task
per task
Why Appen decreased hourly rates almost in all markets after acquiring Leapforce?
This project is about a year old and was created and started when Google decided to exclude some YouTube video evaluation tasks from its main search engine evaluation projects: You are probably familiar with these projects: Yukon (Appen/Raterlabs) and Internet Assessor/Rater at Lionbridge. The Internet Safety Evaluator project picks up where search engine evaluation leaves off and the job has you looking at video search results to improve the search result quality for content. As an Internet Safety Evaluator, you are also required to provide feedback and analysis on video content found in the search results and provide ratings on their relevance with the search terms that were used.
So where does the Internet Safety Evaluator part come in? Well, that’s the most important part and aspect of this role. While reviewing content, you will flag potential upsetting or offensive material to make the internet safer and more secure for all users. You will be able to express your opinions in comments about the video concerning the content and the quality of content, and at the same time, you will be protecting users from viewing unsuitable material.
So what do you need to have to qualify for the position of Internet Safety Evaluator? Appen and Lionbridge are looking for dynamic and creative people who have a strong interest in improving online search results. One of the main things that you should feel comfortable about is working on sensitive or adult content. Candidates must use a smartphone for this position, and either an Android or iPhone device is suitable. The Internet Safety Evaluator should also have some familiarity with social media and Google products. Be prepared to be flexible concerning working hours and you should have the ability to understand and follow the established guidelines.
The Internet Safety Evaluator position only allows you to have 20-hours max work time per week.
The qualification process is identical to the Appen/Yukon Project or the Internet Assessor for Lionbridge. Read more about how to pass that exam here:
Also, don’t forget to check out the really cool tool that we’ve created for helping to pass these tough exams. See Our Exam Simulation Tool Here: https://examsimulation.passleapforceexam.com
Rating Process And Quality Metrics
As an Internet Safety Rater working on the project, you will have access to a variety of different task types. This is important because it helps you become more familiar with the core concepts of rating and it will help you build experience and knowledge concerning the program. During the beginning, the number of tasks you will receive will be low until you gain enough experience in the program. Once you feel that you have a good understanding of the program and become familiar with all the task types, you will start to see a switch in the task types that you receive. But keep in mind, this will only happen if the managers feel you have a strong grasp on the concept of what you are doing. From that point forward, you will only be required to focus on evaluating more video type tasks from an Internet Safety Rater perspective. You will also see a higher number of tasks given to you.
As an Internet Safety Rater, you cannot work more than 1 hour per day and this limitation can last several months. The purpose of this limitation is to evaluate the overall quality of ratings that you are providing. During this time you will be given standard Yukon/Internet Assessor tasks (Needs Met and Side by Side and other types of tasks), additionally, you will be evaluated on your quality by means of standard Needs Met and Side by Side quality review tasks.
If you want to know more about this process, read here:
The quality requirements are high and you need to achieve and keep a quality level higher than 65% or above. This is the very minimum and work is monitored on a regular basis. Feedback will be shared with you so that you can learn from your mistakes. If an Internet Safety Evaluator constantly performs below that minimum quality level or performs significantly lower than the minimum level, that person will be removed from the program. This is why it is very important to use the months long limitations and make sure to learn as much as possible.
The things closely monitored to evaluation an Internet Safety Evaluator performance is the Needs Met, Page Quality (E.A.T.) and SXS rating. However, other things such as rating accuracy and the quality of the comments that you leave are another thing that is closely looked at.
Once the Internet Safety Evaluator has shown a stable level of quality, the probation period ends and you will get a notification that the Internet Safety Evaluation transition period ended and all restrictions are then lifted and you see a higher volume of tasks. This is when you will start receiving ONLY video evaluation tasks and you’ll get a higher volume of them. There will be no more review tasks after the completion of the transition period.
Internet Safety Evaluator probation and transition period
The Internet Safety Evaluator probation and transition period length all depends on the quality performance of the candidate who should demonstrate positive quality review scores. This period usually takes about 3-4 months but could possibly last up to 8 months depending on the performance of the individual.
In terms of the complexity of this project and how easy it is to participate, the Internet Safety Evaluator project seems to be easier in particular after the transition period has ended. Working on video tasks is a lot easier than doing the Needs Met and Side By Side tasks. However, the big drawback to the project is online being able to work 1-2 hours per day during the transition period.
You can work on other projects at the same time to make up for these small hours. You can provide similar rating services to other providers. However, due to project restrictions and customer requirements, you cannot work on Appen Yukon and the Lionbridge Internet Assessor at the same time due to an IP address conflict.
If you want more information on other unrelated projects, you can find that information here:
If you have any questions about the Internet Safety Evaluator position or any other questions, let me know in the comments or simply send me an email and I will reply. If you are working as an Internet Safety Evaluator we’d like to hear about your experience.
If you’ve landed here, you might have heard about the Preferred Raters Program…and It looks like there is finally some great benefits coming out for Lionbridge/Raterlabs for raters who achieve high scores and have consistently high-quality ratings and comments for all task types. This is for raters who work on the Yukon/Raterlabs project. Through this program, agents are identified and rewarded based on high-quality review scores and high-quality work they do on the Yukon project.
It sounds really great, but it is highly selective and prestigious, and only available to a limited amount of raters. The requirements to get into this program are very high, as well as inclusion and retention…meaning it will be tough to get in and also stay there. Participants are held to strict standards of quality and professionalism and the quality team reviews candidates quarterly to determine if they qualify for the program for admission or if you’re already in the program if you stay or someone comes along and takes your place.
So, what about the rewards and are they really worth it?
If you do get into the Preferred Raters Program, you’ll get a boost in pay rate. You’ll go from $10 to $15.00 dollars for Raterlabs and from $14 to $18 at Lionbridge. Not a bad little pay raise at all even if it is temporary.
As another added benefit, your username will be in green in chatrooms showing that you’re a highly skilled and very experienced rater.
How Do I Qualify?
As we mentioned earlier, raters are regularly reviewed to see who meets the requirements of the Preferred Agent Program.
Note: Sending them emails might be the quickest way to NOT get into the program. It is not recommended to email them about your status in the program.
Only if you have met the program’s requirements will you hear from the project’s Quality Team. If you don’t hear from them, then it may mean that you need to step up your quality or performance, or there just may be too many candidates and not enough room to fill them all. We do believe it might be the opposite since requirements are so high for the program.
The most important thing to focus on is quality review scores for all task types. Having high scores in this area will get you looked at for eligibility for the Preferred Raters Program. But it’s not all as there are several other additional criteria used to determine an agent’s final acceptance into the program and we’ll go over all the things they use to select someone.
Primary Selection Criteria
The very first thing to point out is only agents who have worked on the Yukon project for at least one year are eligible.
And agent needs to complete enough of the following tasks to receive the minimum Quality Review feedback:
Needs Met: 6 sets of feedback
Side by Side: 3 sets of feedback
Page Quality: 2 sets of feedback
The quality reviews all have to be extremely positive for all the task types, not just one, and high adherence and demonstrated comprehension of the principles detailed in all the guidelines and project instructions.
Comments are also a part of this and Preferred Raters will show excellent commenting skills for all tasks.
An agent will have a consistent history of good interactions with the community that should always be positive. This includes all areas of communication with admins in chat, social posts, and emails.
While we’re on the subject, we shouldn’t have to tell you that candidates that have a history of unprofessional, nasty, or obnoxious behavior in chat or other social channels won’t be considered for this program.
Agents in the program will always meet the minimum hours’ requirement and complete all the tasks in a responsible manner and not releasing them in an excessive or malicious way because maybe the agent doesn’t want to rate the produced results.
Did we mention that it’s hard to get into the Preferred Raters Program? We should mention it again and also state that it’s not an easy thing to achieve or maintain. But if you give hard work, perseverance, and show a very thorough understanding of the concepts from the General Guidelines, then chances are good that you’ll make it as a candidate.
I’ve made it as a candidate, now how do I keep my status?
We mentioned that the Preferred Raters Program is considered prestigious, highly selective, and it is limited to a small number of raters. That just means it’s going to be a lot more difficult for you to get in and stay there. But after you are in, a rater must maintain the high-quality standards across all task types and have high-quality reviews after acceptance.
If a rater fails to maintain high standards of quality, then they are put into a probational period and given the opportunity to improve. That’s good news since if you are in this program, you’ve worked hard to get where you are. However, if an agent continues to fail meeting the program criteria, they will lose their preferred status and the rewards associated. Also, once you are in the program you can be removed at any time at the Quality Teams’ discretion even without a probational period.
So what if you are put on probation?
If a rater fails to meet requirements they are placed on probation. This probation is actually two more reviews and if your reviews are bad, then you’re out. One other thing: If you are placed on probation and receive a substandard or unacceptable review score in ANY review type during your probationary period, your Preferred Agent status will be immediately removed.
So what triggers probation?
When an agent gets three bad scores within the most recent 6-month period below the Preferred Agent standard, this will trigger a probationary period. Also, if the majority of your recent scores are all bad and not meeting the high standard, you are placed on probation. The bottom line is that your scores should be primarily Exceeds Expectations or higher. Meets expectations do not cut the cake. While they are respectable, they are not high enough scores to be in the program. Multiple scores of Meets Expectations can result in the loss of your Preferred Agent status.
Another thing that should be mentioned is that two Substandard and Unacceptable review scores within a recent year’s time period will trigger a probation period. Any additional scores of the same type will result in the immediate removal from the program. Preferred Raters cannot have such low scores.
The most important requirement of a Preferred Agent is that they consistently provide exceptionally high-quality work and if your scores fluctuate, it might suggest to the Quality Team that you are unable to maintain the high standards of the program, and you will be placed on probation. While it is a disheartening thing to be placed on probation, this does mean that you are being given the chance to improve your quality of rating. Remember that Meets Expectations is not good enough.
Some other things that will provoke the Quality Team are poor commenting habits, excessive releasing of tasks, and complaints about unprofessional behavior in chat or other social channels could lead to probation or it could just see you leaving the program altogether. Remember that we said they have total discretion and can release you from the program at any time for any reason.
If you are put on probation then it’s lucky you and you’ll have to use the opportunity to correct the problems and retain your Preferred Agent status.
As we touched on a little earlier, Quality Teams review agents on a quarterly basis and if there is room to promote new agents, they will do so. If the program is full and all those raters are keeping their high standards up, then there won’t be any new raters accepted into the program. When a rater is removed from a program not meeting high-quality standards, then another rater who is consistently doing this is taken into the Preferred Agent program.
You might be wondering if one low score will put you on probation. The answer is no and the Quality Agents recognize that sometimes bad things happen. However, if you continue to get low scores, then you will be put on probation until you fix them and they only give you two reviews after this and you must score high on BOTH reviews, not just one.
Just so we are sure we covered this, while you are in the program, three low scores will get you noticed by the system which will trigger a probation period.
A rater must meet high-quality standards in all areas and all tasks, not just one area. Because the program is considered prestigious, it is limited to a certain amount of raters. That said, if you are in the program and have some problem in one area, it will show and you will likely end up on probation.
You can earn the status back if you lose it by demonstrating that the quality of your work meets the program’s qualification standards on a consistent basis. Keep in mind that just because a rater has been in the program before, preference will not be given over them when determining new rater eligibility. There are only two times you can be put on probation meaning that the third time and you are out of the program.
We hope that we were able to answer your questions about the Preferred Agent program. Are you in this program? Tell us your experience in the comments and if you need help you can always reach us by email.
There’s another project to work on over at Appen called the Appen Connect Wouri Project. We’ll go over the project a little and briefly touch on the main points, what type of job it is and what requirements there are. We will also touch on the grading process and give a few examples.
The goal of the Appen Connect Wouri Project is to understand the user’s intention as represented by the query, and then determine how good the result is and if it satisfies the user’s needs. Users are using Safari and Spotlight to search for different information needs. Some examples include wanting to find out knowledge or information on something or somebody, wanting to know if there is a certain store nearby, the latest news related to a topic, or since they are MAC users, getting an application on iTunes. This project is a bit like the average search engine evaluator, but it does have a different grading system. However, it uses some of the same ideas from those programs on search engine results.
A desktop or laptop computer is recommended to work on the Appen Connect Wouri Project over a mobile phone and all graders should be checking search results with Safari. Safari can be used on Windows operating systems, but it should be mentioned that there is no support from Apple when using Safari on MS Windows. You might be better off with a MAC computer, but it isn’t required.
To participate in the Appen Connect Wouri Project you have to pass an initial assessment and achieve a minimum score of 80%. If you don’t pass the first time, you’ll get another attempt. After you pass the initial assessment you will be given a final assessment in which you must also pass with a minimum score of 80%.
Wouri Project rating Logic
When looking at a user query, there are three steps that you will take in the grading process.
The first thing is understanding the query. You should make sure it’s not too vague, inappropriate, or in a foreign language.
The second thing is understanding the result. Determine if it is GEO or Time Sensitive. There is also the option to choose both or choose neither GEO or Time Sensitive.
Once you understand the query and the result, you will then decide on a relevance rating for the result based on the query intent.
It is important to note that Appen expects local graders working on the Appen Connect Wouri Project to judge the search results of the same locale. What that means is you must in the same location as the project requires. In this case, you may need to configure your computer and your search engine to be in the language of the project that you are working on.
A web relevant search is one that satisfies the user’s information need. Of course, a search result can be more or less relevant based on how well or how completely it satisfies the users’ needs. It is important, and we cannot stress this enough, that you have the guidelines open and you have thoroughly read them.
In step number one, you need to determine what it is that the user is searching for and decide if it’s valid. You should read the query, and review the third party results and think about the possible user intent that it represents. This is called ‘Understanding the Query.
For example, the query ‘Samsung’ could have many different user intents such as shopping for Samsung products, getting Samsung technical support, or the user may be looking for the Samsung homepage. However, if the user simply types in Samsung, then the query can be too vague.
If you have no idea what the user might be looking for, then the next step would be to go to a popular search engine and study the term. Some common terms could have some special meaning. However, if it doesn’t have some special meaning, or it has too many special meanings with each one having a small portion of intent, then you would select Query is Too Vague.
You might also run into a situation where a query is inappropriate and it encourages illegal behavior, includes profane words, using derogatory language, racial slurs or hate speech. It also includes graphic violence or the query might like to retrieve grisly content. You are not expected to do anything about these types of queries except to mark it as inappropriate.
When in doubt about any type of query, you should check results from other search engines. It’s absolutely crucial to understand what the user is searching for, and you may need to check results from other search engines. The good thing about this project is that top results from other search engines are provided for you. There is a small link that says ‘Web Search Link’ that will automatically do this for you and show you the top results in order to better help you understand the query.
It is VITAL to point out that opening the webpages is a MUST when you are working on this project. If the results match the first result of the other search engine, then the result is likely an excellent match. Make sure to check the results that are returned by the Web Search Link. Each result has a clickable URL, a small description, and a photo where applicable.
We touched a little on GEO and Time Sensitive in the beginning, and since you might not understand what that is, we will go over it to help you understand it better.
If a query is about a store or some location that exists in other locations aside from the searcher, let’s say a Starbucks, or a query like ‘movie theatres near me’ then you know its geo sensitive meaning that it matters where and how close the result to the searcher is. If the searcher is from California and looking for a Starbucks and the first result is the homepage or another location, then it doesn’t satisfy the user’s needs.
Anything that deals with a query that involves some kind of time frame, such as ‘Buy Superbowl Tickets’, age of Harrison Ford, or checking on the weather are results that should be marked as ‘Time Sensitive’.
It is important to understand that a query can be both Geo sensitive and Time sensitive, and a good example is a query such as ‘tickets to the local basketball game’, local movie showtimes, etc.
Once you understand the query and you judge the results, then you will assign a relevant grade for each result. There are 9-grade levels:
● Excellent
● Good
● Fair
● Bad
● Unavailable
● Query In Foreign Language
● Result Page In Foreign Language
● Vague
● Inappropriate
There are many examples in the guidelines that go over difficult queries in order to help you evaluate what results would satisfy the informational need for the searcher. It is a must that you go over the guidelines and read them and understand them so you know how to grade something.
The guidelines go over when you should choose an excellent result, and the biggest point to keep in mind here is to believe that the result will be everything that the user will need, and they will not have to continue searching. If you believe that the result isn’t good enough, and you would also do an additional search, then you might choose ‘Good’ or ‘Fair’ as the grade.
A ‘Good Result is one that the user wants to see above the fold on the device screen. It is typically the first 4-5 results. However, it is not the best possible result, which would be excellent, but it still provides good information. The guidelines have many examples of what should be chosen as a ‘Good Result’.
When choosing a ‘Fair Result’ for the query, the result would only be somewhat relevant, but not be of interest to all users. An example of this would be a query for a celebrity name, but the result is an obscure person that has the same name.
Choosing ‘Bad Result’ would pertain to a result that does not satisfy user intent and is something that should not be shown to users at all. An example of this would be something that we previously mentioned before if a user was looking for a Starbucks Coffee and lives in San Francisco, but the map result is for a store in San Diego. There are other examples in the guidelines.
It should be straight-forward of when you should choose ‘Foreign Language’. If the given query is in a different language from the specified one, or the target language that you are assigned to work, then you would mark this as “Query in Foreign Language”. If you are not sure if the query is in a foreign language, then just search that query to understand the word or words.
Sometimes in the given search results for a query, the URL will be inaccessible, or website link will not be available or a dead link. If this happens then you would grade it ‘Unavailable’ This also includes landing pages that take longer than 10 seconds to load, or you may see that the domain is up for sale by being sent to a redirection page. This grade also pertains to when video content is unavailable or blocked.
There are some other specific result types that the guidelines go over including ‘Ambiguous Queries’ or when the interpretation is much more popular than others. For example, the query is ‘Golden Retriever’ but the result is a song titled the same. This should be graded as fair because the dominant interpretation would be for the dog breed itself and not a song title. There are some instanced when multiple interpretations can be as equally popular though. In order to assign something to the excellent category, it’s a good idea to take into consideration what geo-location the seacher is looking from. Example:
Query is “UM football”, and the result is the official website for the University of Miami football team. If user’s location is in San Francisco, grade the result as Good, since UM could refer to either U of Miami or U of
Michigan and we don’t know which is best. But if the user location is in Miami, grade as Excellent.
As always, if you have any questions about the program, please send an email or reply in the comments. If you are working on this program, then we would love to hear about your experience in the comments.
Ads Rating Project massive timesheet disapproval: on purpose or unprofessional approach?
It appears that lately there are a lot of problems with the Lionbridge payment systems that has resulted in a timesheet disapproval and it’s not actually a new thing. According to comments of my readers and discussions on https://www.reddit.com/r/WorkOnline/comments/9b0n2n/lionbridge_scammers/ (2018), https://www.reddit.com/r/WorkOnline/comments/cvol9y/lionbridge_scammers_2/ (2019, August),https://www.reddit.com/r/WorkOnline/comments/d95mwc/lionbridge_scammingasking_to_remove_hours_again/
The problem with the timesheet disaproval started to happen in May-June 2018. It just seems that now it’s becoming worse and there are more and more raters complaining that the time sheets that are being submitted are being returned. Lionbridge is asking people to carve a lot of hours from their time sheets and it’s obvious that there is a big problem somewhere, but the company isn’t acknowledging that there is an issue, or perhaps they are just doing this to save themselves a lot of money and get away with it. And the raters who are affected by this timesheet disapproval payment problem aren’t being asked to take an hour or two off, in fact, in some cases, people have been asked to cut 30-40 hours from their time sheets, some of them are asked to remove all hours from July timesheet. This is huge and it’s a lot of money for someone. In fact, it appears that it’s happening to a lot of people as one rater who has been working for the company says that he has 6 friends who all got the same notifications. In one case, one of our readers PrestigiousExplorer says that after working one year for Lionbridge the same thing has now happened to him and the company wants him to deduct 68 hours!
The problem on their end which results in the timesheet disaproval appears to be that the hours that are being submitted don’t match their data. What it means is that the amount of tasks completed isn’t matching up with what your submitted amount is. But here’s the thing; if suddenly the Lionbridge payment department starts seeing a lot of discrepancies in time sheets and the data that they have, shouldn’t they realize that there is a problem somewhere? Now suddenly here you have a bunch of people being asked to shave a lot of hours off their time sheets, and even people who have been working for the company for a long time and then suddenly are having issues. It seems that Lionbridge doesn’t care about any of this and are sticking to their guns and won’t budge on time sheets. If you were subject to a timesheet disapproval then you might be stuck with it.
The big question is does no one at Lionbridge see this as a problem? Or is there something else going on?
As you may know raterhub automatically records the time and the number of tasks completed by each rater on the platform. It is very unlikely that the problem takes place there, however many long-term raters remember that raterhub used to have built-in task counter which was very helpful in terms of recording completed tasks, but for some reason this useful tool was removed from the raterhub functionality.
It looks like the Lionbridge system is just an Excel spreadsheet which corresponds timesheet data with data from raterhub (time recorded/the number of tasks completed) and the problem is there. Many raters worked lots of hours in July and got an email with 0 hours/0 tasks recorded by the system. Lionbridge seems to be very unprofessional in the handling of this problem. It undermines the project and the company’s reputation.
But the fact that no one has caught the problem with timesheet disapproval or made any statement about this timesheet disaproval problem since it started happening in July leads one to believe in some kind of conspiracy here. Some people are calling Lionbridge scammers because of what is going on. If you’re suddenly asking a lot of raters to cut hours off their paychecks, there’s an obvious problem somewhere.
It does appear the company had the same kind of problem with timesheet disaproval in May in which they sent out incorrect data. It’s quite possible that this is the same issue, but instead of trying to figure out what is going on, it looks like the raters are getting the short end of the deal.
A company can’t just suddenly take away your pay when you put in the hours and did the required tasks to the best of your ability and as per your contract. There are some laws that protect freelancers and independent contractors, and if you are doing the work, then you should be paid. But for freelancers and independent contractors, your options are limited depending on what country you live in.
Real timesheet correction required due to low productivity vs. Lionbridge error
In some case raters might be asked to correct heir timesheet due to low productivity and it is OK, keep in mind that your average productivity rate should be 1.5 – 2 minutes/per task.
In this project, most of the tasks do not have any recommended time limits. Raters have to manage their own productivity and their own time. The customer does not indicate average estimated time because the priority is quality in rating the tasks rather than productivity, or how many tasks you rate. Raters have to keep in mind that the productivity and time invoiced should be within reasonable metrics. It is OK to spend more time in the beginning and increase your productivity when becoming a more experienced rater.
Appen and Lionbridge have productivity metrics called RPH. It is assumed these metrics are based on the average raters’ performance in the project which is calculated as the ratio between the number of tasks and the time spent within an invoice period completed by all raters. In my view this metric is not useful, because in many cases raters get different types of tasks which require a different amount of time to complete, therefore the RPH metric is not accurate at all. This can lead to a situation where if your productivity doesn’t fall within their RGP metrics, your invoice will be returned for correction and they will ask you to align your RHP to average even though you spent a reasonable amount of time for completing more time-consuming tasks. The customer understands this issue. That is why they do not indicate the average estimated time. But it would seem that Appen and Lionbridge have again invented their own metrics to pay less to its raters.
What raters can do in this situation?
To calculate the number of tasks completed weekly and entered into time sheet vs. Lionbridge records, if you have a huge discrepancy (like most of raters have) this is a clear indication of wrong data sent by Lionbridge.
Send an email to Lionbridge explaining the discrepancies.
If necessary as a proof of working hours/number of tasks a browsing history can be sent, it can be found, filtered by dates and website here: https://myactivity.google.com/myactivity
Additionally I recommend sending an email directly to the Ads rating project admin (Google Ads Eval: ads_eval@google.com) to make them aware that this is some kind of issue that really needs to be resolved.
Do not correct time if you clearly identified discrepancies until Lionbridge responds to your email and provides you with correct data
Perhaps if the right people receive a large response on the issue, someone will take a look and fix the problem. But as of right now things aren’t looking too good. For those of you who have lost wages, hang in there and hopefully, someone will get this straightened out.
If you have questions or comments then leave them below.
Leapforce/Appen/Raterlabs candidate? If you get a job at one of these places, you’re going to need Payoneer account because that is the only payment option available for these companies. Simply sign up for Payoneer under my referral link and you’ll get the simulation tool subscription fee back as Payoneer subscription 25$ bonus once your account is loaded with 1000$! Normally it takes 1-2 months to make 1000$ for EU/USA/Canada raters and 2-4 months for raters from another countries.
