In this post I will discuss some common issues that new raters face. If you have other issues, feel free to share by leaving a comment below.
What kind of tasks to expect while providing search engine evaluation service for Appen/Lionbridge?
If you get a position as a search engine rater with Appen or Lionbridge (Internet Assessor), you may encounter many different tasks. Some of these tasks are your normal rating tasks, and some are side by side, and you may even run into all kinds of experimental tasks. Some of the tasks are easier than others, like experimental tasks. Each task has a different set of instructions, so it’s best to make sure you know exactly what you are doing, and read very carefully each of the task instructions.
Yukon Project Side By Side tasks
Side by Side rating tasks are very interesting and have everything to do with Google algorithm changes. The human role or ‘rating’ in Side By Side tasks is to determine which side, left or right, is better than the other. In this task you are shown blocks on the left and right side. Each block has a slider from Fails to Meet to Fully Meets, and you should choose which one is best for each block depending on the user intent and query. The same guidelines apply for the blocks as they would for a page rating. Meaning that if block meets user intent, it should get the appropriate rating. Once you have chosen the ratings for each block, you will then be asked to choose on a scale which side is better than the other. A comment may be required to give your own opinion on why you’ve chosen which side, and why it is the best over the other side. It’s very important that you base all your ratings on user intent and taking their location in mind. You will also have a quality review on Side by Side tasks once in two months.
Yukon Project Page Quality tasks
This is the main rating task that a rater is hired to rate. You will be given a website URL along with the first 5 questions which check things like foreign language, which if you’re doing English/USA ratings, would need to be in English to be understood from your location. Pages that don’t load need to be checked, as well as flagging for porn.
The second question is asking you to describe the purpose of the page. The purpose of the page and the user intent are entirely different aspects. Make sure you’re checking the website out to see if it matches user intent, and by looking at the website, you should be able to understand what its main purpose is. If the page is malicious, harmful, or just lacks its purpose for what it is searched for, then you should mark that here. Pages that are simply existing to show ads, without any help to the user, should be flagged. And lastly, it is important to make sure that you understand if the page is a YMYL (Your money or your life) webpage. Understanding this is important because it will change the overall rating of the website.
Once you have completed this, you will then rate the different aspects of the page including its main content quality, and the amount of content. Expertise, authority, and trustworthiness, who is responsible for the website, the supplementary content, page design, layout, and website care and maintenance are all graded on a scale from lowest to highest. The last thing to do is to give the website its overall rating. Usually these types of tasks are allowed 9 minutes to complete. A quality review is done each month to see how you rate. So it’s important to make sure that you are taking the time to evaluate each website properly.
Appen Yukon and Lionbridge Internet Assessor Needs Met tasks
This type of tasks you have already saw while doing part 3 of the qualification exam. Each task contains 8-12 result blocks. In each task you will be given a certain query, and a list of search results blocks with links where you have to give a usefulness rating to search results using a scale from ‘Fails to Met’ to ‘Fully Met’
Other Experimental tasks
Sometimes during rating of side by side, you will see experimental tasks pop up. In most cases experimental tasks are standard page quality or need met tasks, but sometimes you will get new types of tasks. Experimental tasks have a wide range of instructions. It’s important to read the instructions for each task as they will be different each time. Experimental tasks can be very simple, or take a few minutes to complete. They are not always available, and each task can be completely different than the other.
Task Comments
A wide variety of tasks require comments; others do not require comments but give you the option to leave them. While it may seem like comments just put time pressure on you and end up disappearing into a black hole. Your comments help Google analysts understand how you interpret queries and the logic behind your ratings.
Additionally, while block-level comments are not required, they can make Google aware of different experiences raters may have had with a result.
The more information we have the better, so please make sure your comments provide an appropriate amount of information. Here are some basic guidelines for common types of required comments:
- SXS comments always need to include what you think the user intent is, which results are good/bad and why that is, and what led you to prefer one side over another.
- PQ comments should not just repeat grid contents (“All the characteristics in the PQ grid are high, so the overall page quality is High too”) – rather, they should explain why a page’s reputation/EAT/MC/SC/etc. are good/bad and why, giving a more step-by-step walkthrough of why you gave a particular PQ rating.
Appen Yukon Project quality feedback & Lionbridge internet assessor monthly quality review
New raters should not do live tasks immediately after getting access to the task portal. It is highly advised that you hold off on rating live tasks if you do not want to be restricted for quality in the next month. Typically tasks which are available for new raters are quality review tasks on which raters are accessed for their quality. Instead, new raters should focus on doing simulation tasks and self-education. Time spent doing this will really pay off, and you wont be restricted for quality in the future. Also, by doing a certain number of simulation tasks correctly you will upgrade your status in the system, and will be allowed to work more hours.
During your participation in the project your service will be evaluated for its quality by means of quality review tasks. In the beginning of every month, a certain number of tasks are uploaded into the system. Normally there are 10-12 tasks of Needs Met or Side by Side, and once in 3 months PQ tasks to be released as quality review tasks. These tasks are quality review tasks up on which your quality is measured every month, in the end of the month you will get your quality score feedback. In case of raters get substandard quality for several consecutive months, most likely such raters will be removed from the rating program.
If your ratings often stand out of general range of ratings provided by other raters in the project, most likely your account will be flagged by the system automatically and your work will be checked manually for quality .
Read more about this topic here:
How to Work at Lionbridge and Appen For a Long Time and Not Get Fired?
recording and invoicing Working time Yukon and Lionbridge internet assessor project
Time management (recording and invoicing working time) of the work you complete can become a problem if you aren’t managing how much time you are spending on tasks. First of all, you should not sacrifice quality for time. In the beginning you may spend more time than the average estimated time (AET) that Appen or Lionbridge allocate to a specific task, but make sure that in the invoice you put average estimated time that they give you. If you spend less time than average estimated time per a task or set of tasks, be informed that your invoice will be returned for correction, Appen and Lionbridge will ask you to put actual time spent, that is why there is no sense to complete tasks faster than average estimated time. Take your time, do everything correctly and watch your time.
the Yukon Project Extension
Do not forget to log on to the Yukon extension each time you work in the system, otherwise your time spent on doing tasks will not be paid.
According to the vendor policy: any time spent logging on/off, reading, studying, and break times are not billable. Any tasks that are not submitted or are released are also not billable. Please make sure you are not including any of this time on your invoice.
Also keep in mind that consecutive invoices submitted with productivity below 100% may result in removal from the project or you may lose your contract with Appen.
Productivity and Average Estimated Time (AET)
When it comes to the time spent working on a task, the listed AET is meant to represent the average amount of time that a task is expected to take. In general, it is best to complete tasks near the given AET, but we understand that some tasks may take a bit longer than others. If more time is needed to complete a task, we encourage you to take it within reason. The overall goal is to complete tasks within 10% of the given AET. When productivity falls below 90%, a Low Productivity Notice is typically sent out.
Appen Yukon project Invoice Returned for Missing or Incorrect Data
Lots of new raters receive the following emails from vendors:
Your invoice has been rejected for missing or incorrect data. Please make the necessary corrections or provide specific details on the reason for each discrepancy.
Time Discrepancy: For the following line item(s) there is a discrepancy between the hours you have submitted and the activity and work recorded on our servers.
Reported: x hours, xx mins
Unaccounted Variance: -xx mins
What raters should do in this situation?
In order to get invoice approved, raters have to deduct xx unaccounted variance minutes from the invoice, in the future make sure only AET is entered into invoice and the Yukon extension has to be on during active rating.
Payment method for Appen, Lionbridge and Raterlabs
HOW to add Already existing Payoneer account at appen connect personal project page
How to earn additional 25$ by signing up to Payoneer via RefeRral Link
Appen Yukon get locked or restricted for Productivity
Working on some tasks much faster than the AET followed by holding on to a task for much longer than the AET and billing for the full amount on your invoice
Raters are allowed to bill for the AET of a task OR the actual time spent working, whichever is less of the two amounts. It has been discovered that some raters working some tasks much faster than the AET followed by holding on to a task for much longer than the AET and billing for the full amount on invoice, which is a violation of official policy and will not be tolerated.
For example: Rater works 5 tasks, each with an AET of 5 minutes. The first 4 tasks rater completes in 1 minute each, and the 5th task rater holds on to for 30 minutes.
If this pattern continues, invoices will be rejected and raters will be removed from the rating program. Be sure that all invoices are in compliance with policy.
Appen Yukon & Lionbridge project multi-accounts
According to Appen Yukon project policy you can only have one account per IP address, while Lionbridge allows to have one account per household. Those pretty much mean the same thing. There can’t be two accounts coming from the same internet service and this requirement come from Google system requirements.
When it comes to Google rating projects or Facebook related projects, Appen and Lionbridge can find duplicate accounts very fast, and as a result, you will end up losing your accounts and they will be terminated. You certainly don’t want that to happen.
Be informed that it is prohibited to have multiple accounts, without special arrangement. Appen and Lionbridge can find duplicate accounts very fast, and as a result, you will lose both accounts.
There are a few things you can do to get around this problem, and arrange them in way that Leapforce, Lionbridge, and Appen won’t be able to detect.
IP address change and network sharing while working in Appen Yukon and Lionbridge projects
In some cases people may share their network, while working as search engine evaluator, this case leads to IP address conflict and removal from the project. Make sure that you have your personal unique IP address and no one uses the same IP to go online for rating project.
When rating tasks on raterhub platform you may change your IP address many times, it will not affect your project status. Make sure that you do your project tasks in a private place and in a country of your project locale.
Non-completion of a minimum required number of tasks
This issue is very common among new Appen Yukon project agents and may lead to termination of contract. This happens when new agents show no activity in the first month of service. In particular, this can happen if new agents enter the project in the second half of month, and for whatever reasons, they did not complete a minimum required number of tasks. In order to avoid this issue, make sure to show the minimum required activity in the raterhub system.
Yukon and Lionbridge project tasks availability and new rater time restriction
In general, Appen and Lionbridge guarantee at least 2 hours of work per day but task availability may depend on project locale. There might be some days when tasks are available for more than 4 hours. Some locales, like the US project provides raters with 24 hrs/7 days tasks availability.
For the rest market task availability may be limited. You should understand that the system of task distribution is based on a ‘first come first served’ basis and you need to be ready to catch your portion of tasks when they are uploaded into the system. In order to identify time slots when task availability is the highest please look at task completion graph, usually tasks are uploaded every day into the system at the same time slots.
Lionbridge does not provide raters with a task availability instrument, so raters have to identify time slots when tasks are uploaded into the system by themselves.
Appen also has a system that restricts access to tasks for new raters, and those who are undergoing their quality review. To lift this restriction, you have to complete a certain number of simulation tasks or wait till the next quality review, if your quality score is good, the restriction will be lifted automatically.
Appen Yukon and Lionbridge raters get locked for quality audit or restricted for quality
If you receive an email notifying you that your account has been placed under quality review, this is a good time to identify your weak points and improve your rating skills. Below you will find some tips on how to deal with a “Locked for quality audit” situation.
- To speed up the process and to get feedback on your mistakes as quickly as possible, I recommend writing to your vendor and ask why exactly your account has been placed under review. Sometimes the automatic system locks raters accounts without specific reasons. Usually such cases are being resolved within 1-2 days.
- While waiting for detailed feedback from your vendor, proceed with doing simulation tasks in order to improve your rating skills. In most cases the issue is related to Side by Side tasks. Make sure that you understand how to deal with this type of task and that you provide well written comments where required. In many cases, raters are placed under quality review due to poorly written comments.
- Once you receive the feedback email with your mistakes/quality issues, read it carefully and reply back with an explanation on how you will improve your weak points.
- Once your status is moved to “Restricted for quality”, keep in mind that your service is under inspection, thus making sure that you do your best to provide the best possible service.
Appen yukon raters hold status restricted for quality after receiving meets expectations quality feedback
Sometimes some raters do not get their restricted status changed to unrestricted with getting a meets expectations or above quality score. This happens because they were restricted for side by side or PQ tasks quality issues and feedback was released for needs met tasks. In order to lift up all restrictions a rater will have to wait for the next quality review cycle for type of tasks a rater was initially restricted for. If you get meets expectations or an above quality score your account will be unrestricted again.
Have you ever wondered how many hours raters work?
Knowing these statistics can be useful for raters in terms of estimating possible productivity .
The survey is completely anonymous and once completed will show the overall statistics.
Thanks for participating!
The first one is called RaterAide which is a task tracker and timer which keeps track of all tasks and shows the time you are spending on each task. It keeps an accurate record of how much time you worked so you can submit accurate time on your invoice. Your data can be stored on Dropbox meaning that everything can be synced across all your mobile devices.
The next tool that can really help productivity is called The Great Acceptor. This is an extension for Chrome which checks for and can optionally accept tasks. The reason this tool is so good is that tasks are given and taken by raters on a first come – first served basis. Sometimes there can be no tasks, so therefore you would need to sit and refresh your browser over and over until you would see tasks that are available to you. The Great Acceptor automatically refreshes the browser and checks for tasks. You can set the refresh interval from 60 seconds and on up from there. From here you have the option to have the tool automatically accept the tasks that it finds. This is completely optional. There is also an email notifications option so that you can be notified if the extension does find tasks. It’s a quite handy tool for grabbing tasks and will be certain to increase your productivity.